Digital Media Archiving, Storage and Preservation for the Modern Age

TT1

The Irony: Analog recordings made 100 years ago are more likely to survive than digital recordings made today!

The Truth: There is not a “Magic Bullet” solution available today that will work for everyone. The digital formats you have recorded your material on coupled with the amount of digital material you need to archive and digitally preserve will dictate the solution and practices that will work best for you.

 

What is digital preservation?

  • Data backup: Making multiple (two or more) copies of a digital file. The copies should be stored in different geographic locations and on different types of storage media to protect against physical or technical disasters.
  • Verification: Regular inspection of all copies of digital files to protect against media or data transfer failure. A related activity is fixity checking, which verifies that a digital file has not been changed, either intentionally or unintentionally.
  • Migration: Regular transfer of all digital file copies to currently supported media and file formats to protect against technological obsolescence.

If analog materials are stored in a cold, dry environment in appropriate containers, their life expectancy will be extended with minimal human intervention. Also known as the “store and ignore” approach, this relatively passive strategy is not possible with digital media.

TT2

Why is migration required?

  • Digital technologies have a finite useful life, but some are better than others. Simple example of this would be to compare modern SSD drives to the “spinner” drives with moving parts. Those moving parts can fail from mishandling and are just as likely to seize from prolonged lack of use when used as a long-term storage medium.
  • Hard drives fail, peripheral connections change, and software is updated. Perfect modern example of this is the eSATA connection falling out of favor since the release of USB 3.0 and Thunderbolt. Another legacy example, how many readers have a stack of floppy disks or ZIP drives laying around from the 90s…does anyone you know have a working floppy or ZIP drive available?
  • File format obsolescence is a serious problem. Does anybody remember the word processing format WordStar from the 80’s? It’s now very difficult to find a piece of software that will read WordStar. The same can be said of Microsoft Works…a popular consumer format from the 90’s that was discontinued in 2006. Only 8 years have passed since it was removed from Microsoft’s support docket and there is no native support for the format in today’s Microsoft products. Third-party software vendors stepped in with convertor tools that will change them over to modern DOCX files.
  • Transcode older obscure masters to modern popular codecs such as Apple ProRes or Avid DNxHD…these choices need to be periodically reevaluated to conform to modern standards.   Video formats seem to date themselves every 5-7 years. But a select few have had some staying power over the long haul. These are the formats that you should conform your footage to. Avoid native camera formats for permanent archival. The software required to decode the media may be just as obsolete as the floppy disk in the future.
  • Remastering to a tape based system digital video formats with longer shelf life such as HDCAM or DigiBETA tapes has its own pitfalls. Hardware to read the media is likely to become inoperable over time and because of the format falling out of favor, the parts to repair that hardware as well as blank tape stock will become scarce.
  • Less than 10% of documentary filmmakers interviewed in The Academy report (cited below) stated that they regularly migrate their content to preserve work or to maintain access to it. If you are among the 90%, you’re asking for headaches down the road when you attempt to access something you shot 10-15 years ago.

TT3

There are many types of files (also called “file formats”), many variations on individual file formats and many kinds of codecs. Survey respondents (from The Academy Report: The Digital Dilemma) identified 26 different moving image file formats with 15 different codecs, and 15 different audio file formats with 6 different codecs. Many of these formats are natively supported today by operating systems running on popular computer platforms, but some require specialized software and/or hardware. Accessing these files requires at least basic level of computer literacy, and in some cases more technical skills, especially when it comes to long-term access.

Whatever the storage system, access to it is usually via a desktop computer with the appropriate physical and network connections. But digital storage alone is insufficient for preservation purposes. Specific preservation actions must be taken and tracked, with file locations and other information noted. Today tracking can be accomplished using a simple database software tool such as FileMaker Pro or Microsoft Access, or a full-function software tool known as a digital asset management system (DAM or DAMS). Other software tools such as file readers and transcoders (for converting one digital file format to another) are also needed to effectively implement preservation actions.

TT4
A couple really good examples of DAM software can be found in Widen Media Collective and Asset Bank.

What the industry needs…

 

  • Replacement for film as an archival medium: We need a method for archiving digital materials that meets or exceeds the storage performance characteristics of traditional film. Such a system does not currently exist.
  • Standardized nomenclature: As storage archives grow in size, it’s helpful to have a recognized naming convention that is universally understood. The International Standard Audiovisual Number (ISAN) and Entertainment Identifier Registry (EIDR) is a start, but it doesn’t address the whole picture.
  • Collaboration: While the Hollywood community can benefit through collaboration because of their proximity and recognition of a common problem, the independent film community is wildly spread out and have drastically different requirements for digital media archival. There needs to be a non-profit digital archival infrastructure to service the needs of independent filmmakers that aren’t backed by the resources of a major studio.
  • Standardized Archival Format: SMPTE and the US Government’s Federal Agencies Digitation Guidelines Initiative are making strides towards archival standards. But they aren’t there yet, and our industry will have to embrace these standards once they are developed.

What can we do now?

TTico1Hard drive storage – For the low budget independent filmmaker, this is probably the easiest to implement, least time-consuming to maintain, and relatively inexpensive compared to the other solutions. Hard drive storage is relatively future-proof. While connections do change over time, adapters are readily available for older hard drive connections. The downside is that hard drives have a relatively short shelf life of about 5-years, and before you know it you have stacks of hard drives laying all over the place.

 

 

 

TTico2LTO Tapes – LTO tapes have long been used by IT professionals for long term archival of critical data.   They have a proven track record for reliability, have a smaller physical footprint than hard drives, offer a very good cost per terabyte value, and have an excellent shelf life of 10-17 years. However, most LTO systems require an IT professional to properly implement and use and have a high up front cost of several thousand dollars to get the required hardware initially. Additionally, this its probably the most time consuming and complicated solution to retrieve or browse media residing on storage tapes.

 

 

 

TTico3BluRay Disc – Some modern TV stations have turned to BluRay for a storage medium. The initial price point is very low, and judging by the current backwards compatibility of optical media it’s future compatibility looks promising. However, a 25GB BluRay means you’ll be breaking up project material across several discs. This clearly adds a layer of complexity, which will require further management.

 

 

 

 

TTico4Cloud Storage – While storing your media on the Internet may seem like a good solution, there are several reasons why the technology isn’t up to snuff. Mainly, while many broadband providers brag about their download speeds, their upload speeds are absurdly slow. Cloud storage of the occasional CF card of home movies shot on a 5D might be a good solution. However, uploading a days worth of shooting 4K on a Sony F55 is another matter altogether. So unless you are sporting a broadcast TV studio grade Internet connection, cloud storage just isn’t practical.

 

 

 

 

 

The Academy of Motion Picture’s Science and Technology Council authored the aptly named “The Digital Dilemma” Parts 1 and 2. These reports were the primary reference material for this article. You can download your copy of the complete reports below.

The Digital Dilemma 1

The Digital Dilemma 2