External SAS connected arrays function in much the same way as SATA based ones, but with a few advantages, that usually come at a significantly higher cost.  SAS is a full duplex interface, and the command set is based on SCSI instead of IDE, allowing higher performance and throughput.  More expensive SAS arrays also support multipath signaling, for greater redundancy in the supporting electronics. (As opposed to the redundancy provided at the disk level by RAID configurations)  SAS also supports much longer cable lengths, up to 10 meters or 30 feet.  This can be advantagious for quiet video editing rooms, since the disk array, which is usually the loudest part of the system, can be located farther away from the users.

A number of vendors have now begun offering external arrays that interface with the host workstation via a direct extension of the PCIe bus.  This allows all of the RAID functionality to be contained within the array, and gives full speed access to the data as if it was contained within the machine.  Among the advantages of removing the RAID functionality from an internal add-on card, are that it can be attached to a laptop via an ExpressCard, which uses the same signaling protocol as PCIe, and that with addition of a few cheap pass-thru cards, an array can easily be moved between systems.  This is definitely not a hot swappable solution, since it accesses the PCIe bus directly, which is initialized at bootup on most systems.  But if your main edit system has a total OS meltdown at a critical point in your project, it should be much easier to access your data from a different system than if you needed to reinstall the PCI SATA RAID card somewhere else, and allow you use your laptop as a backup edit system in certain instances.

Fibre Channel is by far the most expensive option.  Every part of the system is more expensive, the PCIe HBA cards, the fiber cables, and the disk array controllers.  On the otherhand, Fibre Channel offers capabilities that none of the other storage options really do.  It is a hot swappable interface, running on fiber cables that can extend access thousands of feet if desired, and can easily be networked and shared.  Devices can be connected directly together, shared in an Arbitrated Loop, or all attached to a central fibre switch for simplified management.  It is an efficient and low latency interface, and is available in speeds of 1,2,4, or 8Gb per second, and multiple channels can be combined for higher performance.  Higher speed devices are usually backwards compatible with older hardware, similar to the way ethernet works, allowing you to upgrade your storage network one piece at a time.

Choosing the right storage solution depends on your immediate media needs, your available budget, and the direction you anticipate growing in the future. SATA based solutions offer all of the speed you could need if scaled large enough.  SAS can offer similar performance in a smaller package, but at a higher cost.  Sharing data beyond gigabit network speeds requires a storage system that can interface with multiple computers, but that comes at a significantly increased initial cost.  Investing in Fibre Channel storage is usually only worth the expense if you anticipate the need to share your data on a SAN, either immediately or at some point in the future.  I will examine a few popular shared SAN options in my next post.

I was prompted to begin testing USB3 drives after reading an article on tomshardware.com a while back, and I agree with their assessment that Western Digital has delivered the best USB3 storage products.  I have used many of their 1TB and 2TB MyBook 3.0 drives over the last few weeks, and have had stunning results.  The first few drives even came with PCIe host cards in the box, but they are well worth the expense, even if you have to buy them separately.  I can sustain transfer rates of 150MB/s on a clean drive with Windows 7, and routinely backup a Terabyte of data in two to three hours now.  This used to be an overnight task, using USB2 or Firewire based drives.  Even more impressively, I have gotten nearly identical results from using an ExpressCard based USB3 adaptor in a Dell Precision M6400 as well as from the integrated USB3 ports on the HP 8740w Mobile Workstation.  Western Digital’s 2.5inch My Passport line sustains 90MB/s without needing a separate A/C power source, ideal for making backups onset and in the field, when shooting with tapeless acquisition formats.

These transfer rates are clearly a limited by the speed of the disks themselves, so ideally we will see external drives with two disks running nearly twice as fast in the near future.  One sign that that may not be the case is that when copying for one drive to another on the same USB3 controller, I am only getting 50-60MB/s.  That implies that the original 150MB/s may also be limited by the interface, not just the drive, meaning it may not be as simple as adding more disks to extend the capabilities of USB3 attached external drives.  Also I have occassionally run into an issue with the Western Digital drives where I can’t mount more than one at the same time due to a disk signature conflict.  Hopefully that is just a default software configure issue that will disappear in the near future, but I have been able to get around it fairly easily for now.

Blackmagic has tried to take advantage of this bandwidth potential with a number of products including the Pocket Ultrascope, the Intensity Shuttle, and the UltraStudio Pro.  They seem to be a bit ahead of the curve because I have tried all of these in the last few weeks, and have not been able to get them to work.  I tried them on both laptops and desktops without success, even after applying the firmware and driver updates that Blackmagic suggests.  Most existing USB3 host devices are not capable of the maximum bandwidth that USB3 offers, which is required for any HD Video I/O device to function correctly.  Hopefully the coming months will bring improvements in that regard, but until then I expect Blackmagic will have difficulty finding a strong market for their USB3 based products.

In all of these cases it is apparent that more optimization in the host devices and drivers are needed before we can realize the full potential of USB3 for realtime uncompressed HD video work, which is why there isn’t a large selection of external RAIDs being advertized as alternatives to SCSI, Fiber, or eSATA and SAS.  But until that optimization happens, I will be content with my 150MB/s transfers, which are at least 5 times faster than anything I was getting with Firewire or USB2.  And since a lot of what I do involves onset media management and systems administration for editing workstations, USB3 has already made my life much easier, and was well worth the investment.

Elsewhere on the mobile recording front, I have been using some prototype units that record HD-SDI directly to SATA drives.  We tested them in some pretty extreme operating environments, and when used with Solid State SATA drives, they held up pretty well.  While we weren’t without our share of problems, the units were able to capture some pretty amazing footage when combined with an Iconix camera system.  I won’t post a full review until the creators have had an opportunity to solve some of the issues with the units, that our tests exposed.

Nvidia also released a new high end professional video card, the QuadroFX 4700 X2.  This card has two independent GPUs that can be harnessed together with SLI or used separately to drive 4 separate displays.  The stats are not much more impressive than the current top of the line 4600 and 5600 solutions, so they are really just updating the previous 4500X2 which was made obselete by the new generation of GeForce8 based cards released last year.

Dual 3.0 Ghz 5365 Clovertown Intel Xeon Processors (8-Cores)
8GB Ram (Once I get a 64bit OS)
QuadroFX 4600 Graphics Card (To power my 30″ LCD)
2x300GB 15k RPM SAS Drives (125MB/s each)

The XW8400 is currently one step behind the new top of the line XW8600, but the PCI-X slots, wider 3rd party support, and much lower price, weighted in its favor.  I will be doing some tests and benchmarks in the near future, so stay tuned for the results.  I am also curious to see if the Quadro4600 properly allows fullscreen overlay output in Prospect2K’s CineformRT Premiere mode, fixing the Geforce8 overlay problem.

I got a ridiculously good deal on everything, since I waited nearly a year for the perfect opportunity, but I now have a few extra parts that I am not sure how to best utilize:

I already had all the disks I needed, with a 74GB 10k rpm SATA drive for OS and 4x 500GB SATA RAID5 for Data. My Raid5 is only getting 60-80MB/s with the integrated Intel Matrix storage controller, which is less than I was hoping for, but my data will be secure. If I continue to use the Raid5, I won’t have any place for my new, SAS drives worth $1000 bucks apiece. Alternately I could replace my Raid5 with a single 1TB disk for quantity storage with less security, and use the SAS disks in Raid0 for high-speed capture storage.

So that leaves me with a problem/question that I am looking for creative solutions to:
I need a good external enclosure for either 2 SAS drives, or 4 SATA drives. With SAS, the key factor I would be looking for is speed, and with SATA, the key factor would be security. Drobo comes to mind for the SATA drives, but is there a cheaper option with the same level of security, or better bandwidth?  Anyone know any good ways of externally connecting two SAS drives in Raid0?

Anyhow, export and render benchmark results to come, specifically with Premiere, AE, RedCine, and Cineform, plus a few FPS benchmarks for fun.

Let’s start with SCSI, since it is the easiest to dismiss.  While we are discussing the connection of SATA drives, many of the first generation SATA arrays had intergrated controllers and Raid hardware, and then needed a fast connection to the host.  These arrays were designed to replace much more expensive SCSI drive based arrays, so the target customers trusted the SCSI interface, and already had high end SCSI controllers in their systems.  That made SCSI the optimal connection solution for early SATA arrays.  The SATA Raid controller masquerades the entire array as a single SCSI disk, allowing connection to systems through existing SCSI cards.  With up to 320MB/s of bandwidth, a single SCSI channel can efficiently support 5-7 SATA disks without much impact on performance.  The biggest reason to dismiss SCSI as a serious possibility is that eSATA is a better option for most, and the remaining will be much better served by a Fibre Channel interface, allowing for economical upgrading to a full SAN in the future.

The next step for high end SATA arrays was to replace the SCSI emulation with a much more flexible interface, Fibre Channel.  With up to 400MB/s, Fibre Channel has few disadvantages to SCSI, and one major benefit.  SATA disk arrays with Fibre Channel interfaces can usually be connected to switches, and shared between multiple systems, in a SAN.  All connected systems get direct block level access to the disks, which will almost always be faster and more responsive than sharing through an ethernet network.  With the proper Shared SAN software, these systems can also share the data down to the level of individual files.  For facilities where multiple users do collaberative work, based on the same source data, Fibre Channel is probably worth the added initial investment, even if a SAN is not immediately implemented with the purchased hardware.  The possible extensible use of an array beyond a single workstation should be well worth the increase in price, and as an added benefit, cable lengths can easily be increased enough to keep the noisy array out of what should be a peaceful creative environment.

 There are many products available that share storage directly to an ethernet network connection.  The consumer varients hardly have the performance to support DV editing, let alone anything more demanding.  The higher end options, with prices similar to SCSI and FC do offer some interesting possibilities, but will rarely be the optimal choice for a given situation.  Any gigabit ethernet connection is limited to 125MB/s, and in reality, the achievable performance is usually about half of that.  Gigabit network solutions will not be a solution for uncompressed work at HD or higher resolutions.  10Gb Ethernet would offer the desired performance, but is not currently an economical solution.  If compressed files are used, regualr gigabit ethernet can be used to transport the data in realtime, but I would still argue that arrays interfacing directly to ethernet are not the most efficent solution.  Any similar array directly connected to a workstation through a different interface will give much better performance to that system, and can still be shared on an ethernet network via that workstation.  There will be a performance hit on that station when sharing data to other system, but a network card with a TCP/IP Offload Engine (ToE) can minimize that effect, and the increased performance on that system do to the high speed storage directly attached should more than offset whatever is remaining.  This would involve using an array with one of the other interfaces we are examining.

A recent technology that uses ethernet to transfor data, is iSCSI.  Promoted as having many of the advantages of Fibre Channel SANs, iSCSI gives initiator devices (workstations) block level access to their target device (arrays).  This allows the target device on the network to emulate a local device on the initiator’s system.  The downsides are that maintaining data intergrity on shared target drives, requires most of the same expensive software infrastructure that a Fibre SAN does, and the inefficiencies of the TCP/IP protocol are still present to limit the realistically achievable maximum transfer rate.  If you have to deliver identical data to a large number of systems, and don’t want to spend money on the performance that Fibre Channel hardware can deliver, then iSCSI might be of benefit to you.  These products are targeted at large corporations, and don’t scale down in size without losing performance, and maintaining deployment complexity.  I don’t see this being the solution of choice for most desktop PC workstation professionals in post-production field.

 The next solution is offered in a staggering varietly different solutions, eSATA.  This can be fairly confusing due to the number of variations of this technology on the market.  eSATA is a very flexible standard, but not all implementations will deliver optimal results.  For example, some products support port multiplying to increase the number of drives without increasing the complexity of the interface cables or the Raid controller.  This solution is good for high volume solutions, but will not deliver the same level of performance as direct connection based solutions.  The simplest, professional level, eSATA array will be an external drive enclosure that passes each drive’s data interface directly back to the controller, which will usually be some varient of PCI card, inside the workstation.  This gives the card direct full-speed access to each disk drive, and all Raid processing is done on the controller card inside the workstation.  This will be the fastest and most efficient solution for the cheapest price, and I highly recommend it.  The limitations are the cables which usually have a 6 foot maximum length, and the fact that Fibre channel is easier to share.  But for the independent, budget conscious, single workstation user, this is the way to go.  Eight disks gives you enough storage for almost any concievable independent project, and eight drives should support uncompressed HD if desired, and may even work for 2K with an efficient Raid controller.  Solutions that use port multipliers to connect more drives, will increase storage but not performance, and usually require more expensive SAS compatible controller cards to support the port multiplying.  If you need more than 8TB of storage on your system, these might work well for you.

The most recent development in this area is the advent of the use of External PCI Express as an array interface.  A small PCIe passthru card is all that is required in the host system.  An x4 slot can transmit and recieve 10Gb/s of data, which is 1.2GB/s, and there is much less overhead than most other interfaces.  An x8 slot is capable of twice as much throughput for an insignificant margin cost increase.  With External PCIe, the drive controller and raid processing electronics are contained within the drive enclosure, and the controller has direct access to the disks.  As a result, the array could easily be moved to another system, without having to bring a separate controller card from within the system.  Each system would need an External PCIe bracket, but those are only forth about ten dollars.  Due to the nature of the External PCIe interface, the computer has the same level of access to the controller and its data that it would if those electronics resided on a board contained within the workstation.  Another benefit of PCIe, is that the new ExpressCard for notebooks is based on the same interface.  This allows a simple adapter to connect an External PCIe device to a notebook at x1 speeds (over 250MB/s will be fast enough for uncompressed HD).  Currently I am only aware of two vendors offering soluitions using this technology, CalDigit and Ciproco.  It will be interesting to watch as this technology continues to develop.

So my recommendation is that high end eSATA solutions are the most economical direct attached storage solutions, and can support uncompressed HD if needed.  Larger operations that are considering upgrading to a full shared SAN system in the future will probably find the increased initial investment of Fibre Channel arrays to be well worth the value when they re-utilize the same hardware in their SAN implementation sometime in the future.

The industry has responded with many different solutions, that vary in concept beyond recognition and in price by many orders of magnitude.  The earliest solutions involved video tape, analog replaced by digital recording.  Hard disks were introduced for random access to data, and now those are slowly beginning to be replaced by solid state flash chips.  Since this site is targeted to PC users, we will focus on hard disk based solutions, and the interfaces with which they can be accessed by a media workstation.

Hard disks are produced with five popular interfaces:  IDE/ATAPI, Serial-ATA (SATA), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), and Fibre Channel (FC).  IDE and SCSI interfaces are currently being phased out and replaced by their more capable and flexible Serial varients.  I know little of true Fibre Channel hard disks, but that format is rarely used in this industry.  That leaves only two options, which are now somewhat similar and compatible, SATA and SAS.  With identical connection cables, and both offered in 3.5″ and 2.5″ form factors, it is hard to tell the two options apart visually.  Their interfaces both support 300MB/s, dedicated buses for each drive, and port splitting when that is not required.

The biggest differences between SATA and SAS are performance and cost, which eventually distill down to one issue: size.  SAS disks have slightly more capable and efficient electronics, run fewer platter, with less data, and much higher RPMs and faster I/O and transfer rates.  SATA drives usually have much more storage capacity, lower speeds, and are always much cheaper.  At first glance, high end post production work would seem suited for SAS drives, since moving picture footage requires a higher data transfer rate than almost any other application of computing technology.

There are four other factors, which when combined, weight much more heavily in favor of SATA.  The first is price.  Since the difference in price per Gigabyte is currently so great, and SATA drives are not that different in their design or performance, a few quick calculations will reveal that while SAS disks have higher performance per drive, SATA disks deliver more performance per dollar, regardless of their storage capacity.  Second is that the infrastructure needed to aggregate the performance of multiple disks (Raid arrays) will be required, regardless of which disk solution we choose.  This is due to the fact that HD resolutions and larger require much higher data transfer rates than any single drive can provide (unless compressed, and even then, fast disk access is beneficial).  The marginal cost to increase the number of drives being aggregated will be low in many cases.  The third factor is that digitalized footage requires a tremendous amount of storage space, once again contributing to the need for many hard drives to be combined.  Lastly, most of the popular solutions to improve reliability, do so by utilizing even more capacity, to store redundant information in the form of parity, or straight backups.

These factors, when combined make a strong case for SATA disks, which have higher capacity at the expense of performance per drive.  If we are combining drives anyway, the performance benefits of SAS will usually be negated by combining more SATA drives for less money.  This is a case where quantity can clearly overcome quality in most instances.  As a side benefit, SATA drives usually have much greater capacities.

The only time when SAS may be favorable, will be when there is little need for high capacity, and when there is value to smaller solutions.  Fewer SAS disks are required to reach a given level of performance, and will therefore be more portable, require less power, and frequently generate less heat and noise.  For visual effects, were a few seconds of footage are manipulated at very high quality, or short commercials, SAS may be a more efficient option.

In most cases though, the numbers come down in favor of SATA by along shot.  Let’s imagine a two hour movie, with a 10:1 shooting ratio, giving us 20 hours of footage, and for the sake of example, let’s assume a data rate of 100MB/s.  With 3600 seconds in an hour, that is 360000MB an hour, or 360GB.  20 hours of footage would require 7.2TB of storage.  Add 10% to avoid disk fragmentation, and you need an 8TB array.  With 1TB SATA disks you need 8, plus two more to support Raid 50.  You will have the bandwidth of eight drives, and assuming 50MB/s each for SATA disks, and an efficient controller interface, that is 400 MB/s, more than enough for our 100MB/s files.  10 SATA drives at 1TB currently costs ago $3,000, and the Raid hardware will be required by both SATA and SAS, so it does not necessarily need to be factored in.  Now when onlining a production, not all footage is usually captured, but when you factor in captures, conformed exports, film and video colored versions, testless and texted masters, a 10:1 ratio will not be an inaccurate estimate.  Now I used round numbers, so that if the datarate of your format of choice is higher or lower, you can ajust accordingly. 200MB/s footage would need 20 disks, but could get double performance.  50MB/s footage would only need 5 disks, but could still expect 200MB/s of performance.  Have less footage, I left a 4x overhead in this example with 20 hours of source, but I also used 1TB drives for my calculations.  With 10 hours, 500GB drives show SATA to be even more economically favorable.

Now for a quick comparison to SAS, we start by noting that the maximum capacity is 300GB, and you can expect to pay at least $500 per disk.  Our 8TB example would require about thirty disks, assuming a Raid 50, striping together three Raid 5 arrays of ten disks. 27 data disks is 8.1TB for a cost of $15,000 in drives alone, not counting that it requires hardware for three 10 bay array enclosures instead of one.  From a performance perspective, assuming 80MB/s per disk, you can get over 2GB/s if you want to pay for an interface that fast, but remember that this is all for footage that is 100MB/s.  2GB/s might be good if you want to share it between multiple systems, but with that many users, usually multiple productions will be processed concurrently, requiring much more storage capacity anyway.  By multiplying up and down for different formats, it becomes clear that there is no way that SAS can economically catch up.

So I hope this successfully establishes that SATA disk drives will almost always be the drive type of choice for post-production environments.  I plan to examine the different options for connecting these drive arrays to a workstation or group of systems in my next post.