Shared SANs takes the benefits of having all of your storage interconnected with high bandwidth links, and extends it one step further.  By running special software to synchronize the connected systems, it allows each of the connected systems to access the same the data on the same volume on the SAN, without overwriting each others files or corrupting the data.  Most SAN software is designed to function as a peer to peer solution for smaller installations, (5-10 systems) or with dedicated servers for larger SANs.

As is probably obvious, there are many benefits to having multiple systems sharing the same set of files on a central high performance disk array.  First off, you don’t have to buy individual arrays for each system, making individual systems cheaper and quieter.  All the actual data is stored in single physical location, making it easier to protect and secure it.  With all the data stored on centralized volumes, file management is easier, with a single unified file structure, and you lose the need to duplicate source files across every system that needs local access to them.  This saves time and storage space.  It also makes it easier to make thorough backups, especially in automated form, which makes your data more secure.  On the flipside, the initial investment is usually rather high, and all of your eggs are in one basket.  If the SAN has an issue or problem, your entire production may grind to a halt until the issue is resolved.

Most all SANs use Fiber Channel as their primary physical interface.  Although this in not inherently required, until recently there was no other standard technology that offered that capability.  CalDigit recently launched a PCIe switch product that they claim offers shared SAN capabilities for their PCIe attached arrays.  While the idea is great, currently the hardware is still in a similar price range to entry level fiber solutions, and you still need expensive software to keep the connected systems in sync and prevent your SAN data  from getting corrupted.

iSCSI also offers some of the same capabilities, with block level drive access, but is only a viable competitor in the high end production world when running on 10Gb ethernet interfaces, which are still usually prohibitively expensive at this point.  Running iSCSI over Gigabit ethernet may be a viable solution for certain compressed workflows, but offers few advantages over regular network storage, at the expense of needing separate SAN software to share properly.

There are a number of different software options when creating a Shared SAN.  I am not familiar with every one of them, but the five I describe here should give you a place to start.  They all serve the same purpose of preventing multiple systems from trying to write data in the same spot at the same time, but they use a variety of different methods to accomplish that objective.

FiberJet is the cheapest option, but does not allow true file level sharing.  It prevents overwriting and data corruption by only giving one system at a time write access to any given volume.  On the otherhand, all systems can be given full read access any volume all the time.  This allows you to share source footage and other media with multiple workstations without the waste of having to duplicate the files.  It doesn’t allow you to easily share actually project files, since most apps will require write access, and will usually force you to share your files across a number of separate volumes, making it harder to find or backup your data efficiently.  So FibreJet gives you about half of the benefits of a Shared SAN, as a low cost starting point.

MetaSAN has been available for quite a while now, and is fairly common in PC based post-production environments.  It supports true file level sharing, allowing all of your systems to read and write files on the same volume simultaneously.  It supports standard file systems, and operates as a separate process over IP to keep machines in sync.  It also allows PCs to access files on Mac formatted drives and vice versa.  It requires one of the connected systems to host the server process, to manage the distribution of metadata and synchronization information.  That system does not have to be dedicated to that task, but it can be for maximum performance and stability.  If you use a user workstation, rebooting that system could cause other users to lose disk access.  I have used MetaSAN for many years, and it is an amazing tool, but it has its quirks that you have to get used to.  It has a tendency to freeze up workstations if something goes wrong, as it waits for certain requests to timeout, which can make it difficult to troubleshoot when you are in a hurry. (And when the SAN is down, you are always in a hurry)  On the otherhand, with all of its instability and frusteration, it has never allowed one of my arrays to become corrupted, or for me to lose data, so it clearly performs its function.

HyperFS is a recently released option, primarily offered by Rorke Data in the US.  It has its own proprietary file system, which can be directly accessed from Windows, OSX and Linux based systems.  The base software is priced similar to MetaSAN, and functions in a peer to peer fashion in smaller installations.  But if you have more than 8 systems to connect, you will be required to invest in a full dedicated metadata server and license, which significantly increases the deployment cost.

XSAN is Apple’s shared SAN software offering, currently on version 2.2, and it is limited to OSX and requires Xserve systems as metadata controllers.  As a PC guy, I have no experience with XSan, but it is used by many Mac based post-production facilities.  The underlying technology is based on the last option we will examine, StorNext.

StorNext is by far the most expensive option, but it offers higher performance, specifically for frame based media, than any of the other choices.  Frame sequence based media bog down other SAN software due to the high number of individual files that are being opened, accessed, and closed, in rapid sequence.  Each individual frame requires the same amount of metadata and synchronization data as an entire video file, overloading lower end software options.  StorNext is an enterprise level product with a variety of options and tiers, with versions that support every different OS, and even ones that interoperate with Apple’s XSan.  It is clearly an expensive option, but you are paying for stability and performance, putting it at the core of many DI facilities that have a DPX based workflow.

Shared SANs are one off the most complicated and expensive investments available in the post-production world.  Lower cost network based alternatives are a better place to start, for smaller oragnizations and compressed workflows, until you are sure you need the performance that SANs can offer.  Once you are working with uncompressed high definition video, or 2K frame sizes, especially with multiple users, a SAN will probably be worth the investment.  The effect that they can have on your workflow and level of collaboration is dramatic, making them worth the effort it takes to get them up and running.

External SAS connected arrays function in much the same way as SATA based ones, but with a few advantages, that usually come at a significantly higher cost.  SAS is a full duplex interface, and the command set is based on SCSI instead of IDE, allowing higher performance and throughput.  More expensive SAS arrays also support multipath signaling, for greater redundancy in the supporting electronics. (As opposed to the redundancy provided at the disk level by RAID configurations)  SAS also supports much longer cable lengths, up to 10 meters or 30 feet.  This can be advantagious for quiet video editing rooms, since the disk array, which is usually the loudest part of the system, can be located farther away from the users.

A number of vendors have now begun offering external arrays that interface with the host workstation via a direct extension of the PCIe bus.  This allows all of the RAID functionality to be contained within the array, and gives full speed access to the data as if it was contained within the machine.  Among the advantages of removing the RAID functionality from an internal add-on card, are that it can be attached to a laptop via an ExpressCard, which uses the same signaling protocol as PCIe, and that with addition of a few cheap pass-thru cards, an array can easily be moved between systems.  This is definitely not a hot swappable solution, since it accesses the PCIe bus directly, which is initialized at bootup on most systems.  But if your main edit system has a total OS meltdown at a critical point in your project, it should be much easier to access your data from a different system than if you needed to reinstall the PCI SATA RAID card somewhere else, and allow you use your laptop as a backup edit system in certain instances.

Fibre Channel is by far the most expensive option.  Every part of the system is more expensive, the PCIe HBA cards, the fiber cables, and the disk array controllers.  On the otherhand, Fibre Channel offers capabilities that none of the other storage options really do.  It is a hot swappable interface, running on fiber cables that can extend access thousands of feet if desired, and can easily be networked and shared.  Devices can be connected directly together, shared in an Arbitrated Loop, or all attached to a central fibre switch for simplified management.  It is an efficient and low latency interface, and is available in speeds of 1,2,4, or 8Gb per second, and multiple channels can be combined for higher performance.  Higher speed devices are usually backwards compatible with older hardware, similar to the way ethernet works, allowing you to upgrade your storage network one piece at a time.

Choosing the right storage solution depends on your immediate media needs, your available budget, and the direction you anticipate growing in the future. SATA based solutions offer all of the speed you could need if scaled large enough.  SAS can offer similar performance in a smaller package, but at a higher cost.  Sharing data beyond gigabit network speeds requires a storage system that can interface with multiple computers, but that comes at a significantly increased initial cost.  Investing in Fibre Channel storage is usually only worth the expense if you anticipate the need to share your data on a SAN, either immediately or at some point in the future.  I will examine a few popular shared SAN options in my next post.

Panasonic has released a number of new camcorders.  They have three lines of solid-state recording: AVC-Intra to P2 Cards, DVCPro-HD to P2 Cards, and AVCHD to SDHC cards.  At the upper end, the new HPX3700 and HPX2700 both record 10bit 4:2:2 to P2 in AVC Intra.  Both support “Varicam” variable framerate options at 1080p, and the top of the line HPX3700 model also has Dual-Link 4:4:4 RGB output capability.  In the prosumer market, three cameras that I would describe as variations to the HVX200 have been released.  All have 3 1/3″ CCDs, and have the same basis shape look of the original.  The updated HVX200A has improve optics and sensor, as well as the addition of an HD-SDI output.  The lower cost HPX-170 removes the outdated SD-MiniDV tape option option, allowing only P2 recording.  Lastly, the HMC-150 is similar in physical formfactor, but record to SDHC cards in the AVCHD codec, at much lower bitrates.  The other two options in the new professional AVCHD line are the shoulder mount HMC-70 and the 1 lb, 3CCD, HV30 competitor, the HSC1U.

Panasonic has also announced that a 64GB P2 card will be available later this year, which will double the maximum record time of all their cameras utilizing that technology.

Fibre Channel technology marches onwards, with the release of 8Gb products by ATTO.  Not to be confused with the existing 10Gb Fibre Channel specification used for Fibre switch interconnects.  The new 8Gb client connection technology provides 800MB/s bandwidth per channel.  Since I don’t currently have any application for transfers at that speed, I am most excited about the implications this will have on the prices of current “obselete” 4Gb Fibre HBAs, that I actually do use.

Last on the list, is the item I was most impressed with, but have the least realistic practical application for.  I was stunned to see a true 3D image on a large flat panel TV as I walked by a booth.  I guess Philips released this a couple of months ago, but a totally separate company was displaying what they could do with it.  The display uses lense technology at the individual pixel level to control the projected light pattern, which allows it to display different images when viewed from different angles.  With the ability to send each eye a different image, a depth effect is easily created.  Instead of alternating only two left and right images, the Philips solution displays 9 different angles before repeating the sequence, greatly improving the probability that the viewer will percieve the image depth correctly.  This leads to the question of: where do the nine angles come from?  Instead of the traditional way of producing 3D depth effects by creating two separate streams, the Philips TV accepts a single flat video input, as well as a Z-depth map, and renders the different angles live based on that information.  A Z-depth map can be easily created for animated content, but is nearly impossible to produce efficiently for regular video footage, limiting the potential sources of content.  A technology called Declipse allows multple layer to be input, allowing a full lookaround effect for objects in the forground, which I can confirm, adds significantly to the effect.  Although it can not be easily applied to my stereoscopic style of shooting 3D, it was definitely a very impressive technology to see in action.  I intend to keep an eye on its further development, and look forward to seeing it deployed on a large scale.

Let’s start with SCSI, since it is the easiest to dismiss.  While we are discussing the connection of SATA drives, many of the first generation SATA arrays had intergrated controllers and Raid hardware, and then needed a fast connection to the host.  These arrays were designed to replace much more expensive SCSI drive based arrays, so the target customers trusted the SCSI interface, and already had high end SCSI controllers in their systems.  That made SCSI the optimal connection solution for early SATA arrays.  The SATA Raid controller masquerades the entire array as a single SCSI disk, allowing connection to systems through existing SCSI cards.  With up to 320MB/s of bandwidth, a single SCSI channel can efficiently support 5-7 SATA disks without much impact on performance.  The biggest reason to dismiss SCSI as a serious possibility is that eSATA is a better option for most, and the remaining will be much better served by a Fibre Channel interface, allowing for economical upgrading to a full SAN in the future.

The next step for high end SATA arrays was to replace the SCSI emulation with a much more flexible interface, Fibre Channel.  With up to 400MB/s, Fibre Channel has few disadvantages to SCSI, and one major benefit.  SATA disk arrays with Fibre Channel interfaces can usually be connected to switches, and shared between multiple systems, in a SAN.  All connected systems get direct block level access to the disks, which will almost always be faster and more responsive than sharing through an ethernet network.  With the proper Shared SAN software, these systems can also share the data down to the level of individual files.  For facilities where multiple users do collaberative work, based on the same source data, Fibre Channel is probably worth the added initial investment, even if a SAN is not immediately implemented with the purchased hardware.  The possible extensible use of an array beyond a single workstation should be well worth the increase in price, and as an added benefit, cable lengths can easily be increased enough to keep the noisy array out of what should be a peaceful creative environment.

 There are many products available that share storage directly to an ethernet network connection.  The consumer varients hardly have the performance to support DV editing, let alone anything more demanding.  The higher end options, with prices similar to SCSI and FC do offer some interesting possibilities, but will rarely be the optimal choice for a given situation.  Any gigabit ethernet connection is limited to 125MB/s, and in reality, the achievable performance is usually about half of that.  Gigabit network solutions will not be a solution for uncompressed work at HD or higher resolutions.  10Gb Ethernet would offer the desired performance, but is not currently an economical solution.  If compressed files are used, regualr gigabit ethernet can be used to transport the data in realtime, but I would still argue that arrays interfacing directly to ethernet are not the most efficent solution.  Any similar array directly connected to a workstation through a different interface will give much better performance to that system, and can still be shared on an ethernet network via that workstation.  There will be a performance hit on that station when sharing data to other system, but a network card with a TCP/IP Offload Engine (ToE) can minimize that effect, and the increased performance on that system do to the high speed storage directly attached should more than offset whatever is remaining.  This would involve using an array with one of the other interfaces we are examining.

A recent technology that uses ethernet to transfor data, is iSCSI.  Promoted as having many of the advantages of Fibre Channel SANs, iSCSI gives initiator devices (workstations) block level access to their target device (arrays).  This allows the target device on the network to emulate a local device on the initiator’s system.  The downsides are that maintaining data intergrity on shared target drives, requires most of the same expensive software infrastructure that a Fibre SAN does, and the inefficiencies of the TCP/IP protocol are still present to limit the realistically achievable maximum transfer rate.  If you have to deliver identical data to a large number of systems, and don’t want to spend money on the performance that Fibre Channel hardware can deliver, then iSCSI might be of benefit to you.  These products are targeted at large corporations, and don’t scale down in size without losing performance, and maintaining deployment complexity.  I don’t see this being the solution of choice for most desktop PC workstation professionals in post-production field.

 The next solution is offered in a staggering varietly different solutions, eSATA.  This can be fairly confusing due to the number of variations of this technology on the market.  eSATA is a very flexible standard, but not all implementations will deliver optimal results.  For example, some products support port multiplying to increase the number of drives without increasing the complexity of the interface cables or the Raid controller.  This solution is good for high volume solutions, but will not deliver the same level of performance as direct connection based solutions.  The simplest, professional level, eSATA array will be an external drive enclosure that passes each drive’s data interface directly back to the controller, which will usually be some varient of PCI card, inside the workstation.  This gives the card direct full-speed access to each disk drive, and all Raid processing is done on the controller card inside the workstation.  This will be the fastest and most efficient solution for the cheapest price, and I highly recommend it.  The limitations are the cables which usually have a 6 foot maximum length, and the fact that Fibre channel is easier to share.  But for the independent, budget conscious, single workstation user, this is the way to go.  Eight disks gives you enough storage for almost any concievable independent project, and eight drives should support uncompressed HD if desired, and may even work for 2K with an efficient Raid controller.  Solutions that use port multipliers to connect more drives, will increase storage but not performance, and usually require more expensive SAS compatible controller cards to support the port multiplying.  If you need more than 8TB of storage on your system, these might work well for you.

The most recent development in this area is the advent of the use of External PCI Express as an array interface.  A small PCIe passthru card is all that is required in the host system.  An x4 slot can transmit and recieve 10Gb/s of data, which is 1.2GB/s, and there is much less overhead than most other interfaces.  An x8 slot is capable of twice as much throughput for an insignificant margin cost increase.  With External PCIe, the drive controller and raid processing electronics are contained within the drive enclosure, and the controller has direct access to the disks.  As a result, the array could easily be moved to another system, without having to bring a separate controller card from within the system.  Each system would need an External PCIe bracket, but those are only forth about ten dollars.  Due to the nature of the External PCIe interface, the computer has the same level of access to the controller and its data that it would if those electronics resided on a board contained within the workstation.  Another benefit of PCIe, is that the new ExpressCard for notebooks is based on the same interface.  This allows a simple adapter to connect an External PCIe device to a notebook at x1 speeds (over 250MB/s will be fast enough for uncompressed HD).  Currently I am only aware of two vendors offering soluitions using this technology, CalDigit and Ciproco.  It will be interesting to watch as this technology continues to develop.

So my recommendation is that high end eSATA solutions are the most economical direct attached storage solutions, and can support uncompressed HD if needed.  Larger operations that are considering upgrading to a full shared SAN system in the future will probably find the increased initial investment of Fibre Channel arrays to be well worth the value when they re-utilize the same hardware in their SAN implementation sometime in the future.

The industry has responded with many different solutions, that vary in concept beyond recognition and in price by many orders of magnitude.  The earliest solutions involved video tape, analog replaced by digital recording.  Hard disks were introduced for random access to data, and now those are slowly beginning to be replaced by solid state flash chips.  Since this site is targeted to PC users, we will focus on hard disk based solutions, and the interfaces with which they can be accessed by a media workstation.

Hard disks are produced with five popular interfaces:  IDE/ATAPI, Serial-ATA (SATA), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), and Fibre Channel (FC).  IDE and SCSI interfaces are currently being phased out and replaced by their more capable and flexible Serial varients.  I know little of true Fibre Channel hard disks, but that format is rarely used in this industry.  That leaves only two options, which are now somewhat similar and compatible, SATA and SAS.  With identical connection cables, and both offered in 3.5″ and 2.5″ form factors, it is hard to tell the two options apart visually.  Their interfaces both support 300MB/s, dedicated buses for each drive, and port splitting when that is not required.

The biggest differences between SATA and SAS are performance and cost, which eventually distill down to one issue: size.  SAS disks have slightly more capable and efficient electronics, run fewer platter, with less data, and much higher RPMs and faster I/O and transfer rates.  SATA drives usually have much more storage capacity, lower speeds, and are always much cheaper.  At first glance, high end post production work would seem suited for SAS drives, since moving picture footage requires a higher data transfer rate than almost any other application of computing technology.

There are four other factors, which when combined, weight much more heavily in favor of SATA.  The first is price.  Since the difference in price per Gigabyte is currently so great, and SATA drives are not that different in their design or performance, a few quick calculations will reveal that while SAS disks have higher performance per drive, SATA disks deliver more performance per dollar, regardless of their storage capacity.  Second is that the infrastructure needed to aggregate the performance of multiple disks (Raid arrays) will be required, regardless of which disk solution we choose.  This is due to the fact that HD resolutions and larger require much higher data transfer rates than any single drive can provide (unless compressed, and even then, fast disk access is beneficial).  The marginal cost to increase the number of drives being aggregated will be low in many cases.  The third factor is that digitalized footage requires a tremendous amount of storage space, once again contributing to the need for many hard drives to be combined.  Lastly, most of the popular solutions to improve reliability, do so by utilizing even more capacity, to store redundant information in the form of parity, or straight backups.

These factors, when combined make a strong case for SATA disks, which have higher capacity at the expense of performance per drive.  If we are combining drives anyway, the performance benefits of SAS will usually be negated by combining more SATA drives for less money.  This is a case where quantity can clearly overcome quality in most instances.  As a side benefit, SATA drives usually have much greater capacities.

The only time when SAS may be favorable, will be when there is little need for high capacity, and when there is value to smaller solutions.  Fewer SAS disks are required to reach a given level of performance, and will therefore be more portable, require less power, and frequently generate less heat and noise.  For visual effects, were a few seconds of footage are manipulated at very high quality, or short commercials, SAS may be a more efficient option.

In most cases though, the numbers come down in favor of SATA by along shot.  Let’s imagine a two hour movie, with a 10:1 shooting ratio, giving us 20 hours of footage, and for the sake of example, let’s assume a data rate of 100MB/s.  With 3600 seconds in an hour, that is 360000MB an hour, or 360GB.  20 hours of footage would require 7.2TB of storage.  Add 10% to avoid disk fragmentation, and you need an 8TB array.  With 1TB SATA disks you need 8, plus two more to support Raid 50.  You will have the bandwidth of eight drives, and assuming 50MB/s each for SATA disks, and an efficient controller interface, that is 400 MB/s, more than enough for our 100MB/s files.  10 SATA drives at 1TB currently costs ago $3,000, and the Raid hardware will be required by both SATA and SAS, so it does not necessarily need to be factored in.  Now when onlining a production, not all footage is usually captured, but when you factor in captures, conformed exports, film and video colored versions, testless and texted masters, a 10:1 ratio will not be an inaccurate estimate.  Now I used round numbers, so that if the datarate of your format of choice is higher or lower, you can ajust accordingly. 200MB/s footage would need 20 disks, but could get double performance.  50MB/s footage would only need 5 disks, but could still expect 200MB/s of performance.  Have less footage, I left a 4x overhead in this example with 20 hours of source, but I also used 1TB drives for my calculations.  With 10 hours, 500GB drives show SATA to be even more economically favorable.

Now for a quick comparison to SAS, we start by noting that the maximum capacity is 300GB, and you can expect to pay at least $500 per disk.  Our 8TB example would require about thirty disks, assuming a Raid 50, striping together three Raid 5 arrays of ten disks. 27 data disks is 8.1TB for a cost of $15,000 in drives alone, not counting that it requires hardware for three 10 bay array enclosures instead of one.  From a performance perspective, assuming 80MB/s per disk, you can get over 2GB/s if you want to pay for an interface that fast, but remember that this is all for footage that is 100MB/s.  2GB/s might be good if you want to share it between multiple systems, but with that many users, usually multiple productions will be processed concurrently, requiring much more storage capacity anyway.  By multiplying up and down for different formats, it becomes clear that there is no way that SAS can economically catch up.

So I hope this successfully establishes that SATA disk drives will almost always be the drive type of choice for post-production environments.  I plan to examine the different options for connecting these drive arrays to a workstation or group of systems in my next post.