FEATURES
STORAGE WARS
Companies Vie for the Solution to Saving all that Digital Content
by Michael Kunkes
 G-Technology's G-RAID Firewire 800 atop a Macintosh G5. |
Recently, the post industry has had to wrestle with an intriguing new series of storage and networking technologies that, for the most part, were birthed in other sectors. They range from traditional, general-purpose storage systems in either network-attached storage (NAS) or multiple––drive storage arrays to storage area networks (SAN) and local area networks (LAN). Many of these technologies were evolved to be compliant with federal guidelines for saving documents such as e-mail messages, medical records, blueprints, digital X-rays, mechanical drawings and the like. The explosion in content storage has been phenomenal. According to a recent study by the Yankee Group, digital content has gone from 18 percent of storage-compliant reporting at companies in 2002 to 39 percent in 2006.
Fortunately, the entertainment industry is not bound by compliance, but it has benefited greatly from the technology. Our digital content storage needs require real time access to content, fail-safe reliability without data loss, the ability to work with multiple streams in higher bandwidths, instantaneous media management and collaboration––the ability of multiple users to access the same files in a shared environment. Facilities must consider whether their storage environment will be individual (local) or shared (networked), the amount of storage needed, be it uncompressed or compressed (HD, SD, HDV, DVC-Pro) and, as always, the costs of storage and return on investment.
High Definition (HD) has been driving the storage boom, though compressed HD workflows have lagged behind uncompressed. If anything, HD has dictated at least a quadrupling of the need for uncompressed storage space, a less than ideal situation for post workflows. The future for compressed HD is looking brighter due to the rise of 2K and 4K technologies, digital intermediaries (DI), and new codecs such as the Avid DnxHD (Digital Nonlinear Extensible), which enables HD editing over existing SD infrastructures. “You’re seeing people doing their DI scan from day one, and on bigger films, they are in 2K and 4K land where everything is being downconverted to HD, or even down to SD for the offline edit,” muses
 The Archion Alliance Pro networked storage system. |
James Tucci, chief technology officer of Archion Technologies of
Burbank, California. “Now isn’t that a weird concept!”
For Avid Technology, Inc. (www.avid.com), known more for nonlinear
editing (NLE) systems, storage has become big business, mainly because
the company has figured out how to crack the real-time barrier in
industry-standard Ethernet connections. According to the most recent
available sales figures, storage accounted for $100 million in sales,
or about 18 percent of Avid’s total 2004 revenues.
The company staked out its claim in the shared storage market in 1999 with Unity, which today consists of three entities: 1) Avid LANshare EX, a relatively small local area network that scales up to four terabytes, directly connecting up to five fibre channel users and with a standard Ethernet connection, up to 20 users can connect in real time to one LANshare EX system; 2) AVID Unity MediaNetwork, which scales up to 20 terabytes (Tb) and 60 real-time clients; and 3) Avid Unity ISIS (Infinitely Scalable Intelligent Storage). Unity ISIS offers modular support for up to 100 real-time 50 megabyte clients with simultaneous access to a high end of 64 terabytes of storage (growing to 192 Tb later this year), providing clients with isochronous performance (delivering data uniformly within set time constraints) and guaranteed levels of transport, all over “gigE” Ethernet.
“We launched ISIS for three reasons,” says Andy Dale, Avid’s senior product manager for storage and networking. “Unity MediaNetwork scales up to 20 terabytes, but an increasing number of customers wanted more storage than that. Second, those customers also want the highest possible amount of redundancy and reliability in their storage; and third, rather than have a fibre channel island of production storage that contains their production and is separate from all other networks, they want their production storage more fully integrated into their overall IT infrastructure,” he says. “The advantage of integrating into the IT infrastructure is not just so that editors have access to storage, but so that anyone on the Ethernet network who has permission to access throughout the facility, can browse material, do reviews, approvals and a lot of other things that aren’t necessarily enabled on a fibre channel network.
The ISIS file system was also designed around the concept of distributed intelligence to vastly increase redundancy protection within the system. The drives are part of an intelligent blade that is hot-swappable, as are the power supplies and the integrated Ethernet switching that is built into the chassis. “By adding a distributed intelligence architecture, we’ve taken redundancy to a level that no other storage subsystem has,” Dale relates. “Each of the storage and switching components has redundant images of the OS and the Unity ISIS file system. If there is some problem on one of the images, the system will automatically fail over to the other image and continue to run.
One thing Avid doesn’t offer is full hardware RAID storage, says Archion’s Tucci. As one of the industry’s few pure technology companies, Archion (www.archion.com) commands the largest installed base of alternative storage at facilities and studios, taking the mantle formerly held by the now-defunct Trans-Soft. Archion’s technology is based largely around RAID-based storage (Redundant Array of Independent Discs or Drives), a technology first articulated by a group of visionary professors at Berkeley in the mid-1980s. RAID has taken many forms and gone to many levels, but the most important part of any RAID-based storage is built in redundancy of data, hardware-based protection that is essential for any network-based storage.
Last Fall, in keeping with its philosophy of designing flexible systems for clients that co- exist with their current technologies, Archion introduced Synergy Plus, a product designed specifically to expand and enhance existing Avid Unity systems. Synergy Plus provides hardware RAID protection within a Unity environment, adding storage expansion without doubling storage requirements or incurring additional software expenses.
“If you analyze how Unity works, there is a real beauty to the software,” Tucci says. “The problem is that even though the drives are all striped, they are sort of self-referencing, and you have to hope that when you pull a drive, you’re not going to have the mirrored data in the same space. Syn-ergy Plus was design-ed to respond to the concerns our clients have about the existing Avid Unity storage system, which center on the inherent risk for media loss in the event of a drive failure. We love all of Avid’s products, and we are the only technology company that has engineered a ‘friendly’ hardware RAID technology to expand and complement Unity. As long as you want to have higher bandwidth with protection, you’re going to need RAID storage.”
However, Dale points to Avid’s own proprietary real-time RAID technology, which it calls UnityRAID. “UnityRAID provides the highest level of data protection by writing two exact copies of each piece of data,” he says. “The file systems are randomized such that mirrored information is always kept on separate drives to eliminate any danger of losing both copies by pulling a single drive.” Dale adds that in typical RAID storage, multiple editors accessing data on a specific drive have to wait in line for their data, while editors on UnityRAID would be sent to mirrored copies on other drives.
And while Unity ISIS represents what he calls “a whole new
ballgame” for Avid, Tucci points out that it is an all-Ethernet
system. “It works at gigabyte speeds, and from all indications,
that’s where it’s going to stay,” he says. “In
its current form, that’s just not going to work for uncompressed
hi-def.”
That may be true, Dale concedes, but he adds that plans are in the
works to expand both ISIS’ total storage capacity and bandwidth.
Even still, according to Dale, “ISIS on GigE has a significant
HD readiness and supports up to 40 clients running dual-stream DnxHD
145 for HD editing.”
Synergy Plus is a new addition to a product line that includes network
solutions such as Alliance and Alliance Pro, which are fiber channel
networked storage, and the Solo FireWire single-user drive, which
supports multi-stream uncompressed SD, DVC-PRO HD, HDV and DV NLEs.
Editors Richard Chew, ACE, and Saar Klein recently finished director
Terence Malick’s The New World using an Avid hooked up to Synergy
running Alliance software. Chew also cut on location using Avid Express
Pro on a laptop hooked up to a Solo system.
“Each day’s dailies were given to Richard on a Firewire
drive,” Tucci said. “He copied them onto his Solo Firewire
RAID drive, then e-mailed the edits back to the cutting room.”
Currently, Robert Zemeckis’ Beowulf is being cut by Jeremiah
O’Driscoll and Arthur Schmidt with two Adrenalines, one Meridien
and two Express Pros––all hooked up to an Alliance system.
“Alliance has no metadata controllers, so you’re not using
tons of bandwidth trying to keep track of all that extra data,”
he explains. “It’s just a direct connect pipe to the storage
and full bandwidth.”
 G-Technology's G-RAID Firewire 800 atop an Apple PowerBook. |
Santa Monica, California-based G-Technology, Inc. (www.g-technology. com) has created what every editor seems to want: simple, low-cost, high-performance, cross-platform storage. Founded by Roger Mabon in 2004 and located in the city’s media district, the company is best known for its G-RAID Firewire 800 drive. Designed to support the demands of broadcast video, the G-RAID was designed by Mabon while he was vice president of marketing at Medea. “High-speed disc storage for video was my world at Medea, so I am very much in tune with the needs of that market,” he explains. “G-RAID is basically a simple, direct-attach (Mac or PC), two-disc RAID system, built strictly for speed. The cost is under $500 for 500 gigs, and it can handle a couple of streams of uncompressed SD in real time, as well as all the compressed HD formats.” Mabon adds that G-Tech sells literally thousands of G-RAIDs every month, with industry penetration that includes its use on Martin Scorsese’s The Aviator, as well as last year’s Super Bowl and World Series.
In March, G-Tech will debut G-RAID Pro, a five-disc, Firewire 800 interface, RAID-3 device, the company’s first foray into protected storage. “We’re talking about a terabyte of bulletproof storage for under $2,000, very affordable for what it does,” Mabon adds. “The company is also looking forward to NAB 2006 in April, where it plans to debut a five-drive system with either a SATA (Serial-ATA) or PCI interface that will have the same characteristics as the G-RAID Pro, but with a much higher data rate. “The reason we’ve done so well in this niche is because we identified the weaknesses in other Firewire devices, such as performance deficiencies and overheating problems. We just came up with a product that was very solid and hadn’t been seen before.”
 ProMax's SATAMAX serial ATA array. |
Editors who find all this a bit dizzying might want to pay a visit to Irvine, California’s ProMax Systems, Inc. (www.promax.com), which for 12 years has been in the business of providing branded solutions for film, rich media, sound, content creation and, more recently, shared storage networks. In this area, the company has its own line of affordable stand-alone Firewire drives, disc-arrayed storage boxes, and its SATAMAX line of serial ATA arrays. ProMax specializes in blending its storage products as part of customized, all-in-one systems for non-linear editing, compositing or audio, with full hardware and software support and training. It’s a niche the company created for itself in the earliest days of Avid’s feature film market penetration.
In addition to its Firewire drives, ProMax also supplies storage area network protocols through SANmp, a multi-platform, volume-sharing software, iSANmp with increased I/O throughput, Apple’s X-san high-performance 64-bit SAN file system and Charismatic Engineering’s FibreShare package, which enables out-of-the-box set-up for high-speed SANs. “We take all the technical aspects out of the equation for the customer who doesn’t want to fool with all the technology, says Mac product manager Jerry Miles. We put the system together, install the cards and the software and put each system through its paces for 48 hours before delivery.”
 Texas Memory Systems' RamSan 400 module, shown with hot-swappable drives. |
Solid-state discs are a storage medium that has always delivered fast data movement with high bandwidth, using memory as its primary storage medium instead of hard drives, but its production penetration has been limited due to its higher cost. Houston-based Texas Memory Systems (www.texmem sys.com), founded in 1978, has been pushing the solid-state disc as it attempts to enter the post industry to complement its core business of high-performance databases.
“Our system can move data at 2.8 gigabytes/second, sequentially or randomly, which is very significant if you are talking about non-linear editing” says executive vice president Woody Hutsell. The problems with many disc-arrayed systems, he says, come with multiple editors moving around the same data in the higher bandwidths associated with HD work. “If you have one editor and a RAID system, that’s a piece of cake; you’re just moving data sequentially and that’s the one thing that hard drives are good at,” he continues. “But if you have multiple editors hitting the same content on a RAID array, the performance drops to a tenth of what it was before. Our systems don’t have any of that, so that means that when you scale users on a solid-state, disc-based system, you get incredibly low latency and the advantage of handling very high bandwidths for random data access. The challenge we’ve faced is making it less expensive. Four terabytes of solid state data will set you back a lot of money.”
One solution the company is considering is a version of the disc that acts as a cache only, and not as a storage device. “You can put anywhere from 32 to 128 gigs of cache in front of a fiber channel RAID array, and now if you have multiple editors dealing with one set of data, they can treat that data as cache, and that gives you a lot of performance benefits as a result,” Hutsell adds.
“Editors are busy dealing with the creative process, and it doesn’t matter to them where it’s coming from. They just want the system to work and getting it from us turnkey is a huge assist,” says ProMax’s Miles. “That’s where the the assistant editor comes in. They are more attuned to the technical side. They’re the ones going through the logging and pulling all the content into the system.”
The format debate in storage continues for the immediate future. “The compressed formats, such as DVC Pro HD, are going to be really huge,” says G-Tech’s Mabon. “And it’s only a matter of time before a company is able to supply really inexpensive SAN systems to enable collaboration in the post houses and facilities. We need an affordable shared storage solution that works well with the compressed formats of HD.”
“I think the most challenging situations for editing, storage and networking will involve the uncompressed DI workload, adds Avid’s Dale. “For television, deliverables will continue to be SD and HD formats derived from DI, but sooner or later the uncompressed DI has to come back out, whether it is for theatrical release in another market or simply a long-term archive format for film-originated material. For theatrical, compressed DI formats may be used for offline editing, but effects and color correction will increasingly require real-time performance with 2K and 4K and perhaps even 8K. Eventually, editors will also want to collaborate on shared storage even at these resolutions.”
MORE TO STORE
Ciprico (www.ciprico.com)
In January 2005, Ciprico, a storage solutions provider for content
creation, broadcast and delivery, announced its acquisition of Huge
System, Inc., suppliers of entry-level storage for desktop production.
Proprietary media vault RAID storage technology acquired from the
Huge sale enabled the company to secure a 2005 US patent titled “Media
Server with Single Chip Storage Controller,” which, according
to chairman and CEO James Hansen, is the beginning of a “patent
portfolio” designed to protect its own technologies. In December,
the company also announced the 10G DiMedia NAS, allowing multiple
users to share and view uncompressed SD and compressed HD content
over 10 Gb Ethernet.
Sun Microsystems (www.sun.com)
Last August, Sun acquired StorageTek, leveraging the latter’s
35 years in storage and data management to create a company with combined
2005 revenues of $13.3 billion, with a 36 percent share of the planet’s
archived data. The new entity becomes part of Sun’s Data Management
Group. Mainly designed to support Windows, Linux and Solaris operating
systems, the company’s line of StorEdge products include mid-range,
workgroup and NAS systems.
Glyph Technologies (www.glyphtech.com)
Long a leading supplier to the digital audio industry, Glyph has been
moving more and more into video post. The company’s most recent
productions include four additions to the GT series of tabletop Firewire
400 and 800-based storage, as well as the new PortaGig pocket-sized,
Bus-powered 7200-RPM mini-drive, powerful enough to support high-track
ProTools sessions.
Maxtor (www.maxtor.com)
The Milpitas, California-based company has been drawing accolades
for its new $899.95 OneTouchIII Turbo, a one Tb drive that combines
the disk-striping speed of RAID-O with RAID-1 mirrored-data security.
Pre-formatted for Macs, the drive can connect up to 62 Firewire 400
and 800 devices, and up to 127 USB 2.0 devices through hubs.
Rorke Data (www.rorke.com)
Based in Eden Prairie, Minnesota, Rorke Data has a dedicated broadcast
and post division that is focused on Fibre Channel networks and desktop
solutions. The company’s current product line includes ImageSAN2,
which enables users to modify and leverage workflow by enhancing user
collaboration with improved file-level storage and management; StreamMine,
which facilitates sharing between multiple video servers to vastly
increase available storage; and Galaxy HDX 4GB, a fourth-generation,
low-cost proprietary RAID system designed to exceed current bandwidth
and high capacity storage requirements for the HD market.
BlueArc Corp. (www.bluearc.com)
The San Jose, California-based company recently introduced the Titan
Storage System, billed as the first storage solution that consolidates
and manages up to 256 Tb of data in a single storage pool. Titan’s
scalability allows literally thousands of clients to be served simultaneously,
an idea which has led the product’s penetration into the CGI
and effects studio market.
Michael Kunkes is a freelance editor and writer specializing in animation, production and post-production.
[ return to top ]