Adobe announced all of the features of the new highly anticipated Creative Suite CS5, which will include native 64bit, CUDA GPU acceleration, and better support for formats like DPX and DSLR clips.  I will have many more CS5 details in future posts, delving into how that will change the landscape of many workflows.  There are lots more new developments in the post world, but those are the primary things that stick out to me right now.  We still have two more days, and I will be spending a good bit of time demonstration Cineform’s Neo3D features at their booth at the back of the Lower South Hall.  So if anyone wants to catch up with me, feel free to stop by.

I know I haven’t posted anything here in a while, but there will be quite a bit coming in the near future.  I am writing a series of articles that will be posted between here and Shane Hurlbut’s blog at hurlbutvisuals.com.  I have worked with Shane on a number of projects, the largest one being the Navy Seal film over the last year.  I will be offering an overview of the post workflow options for video DSLRs on his site, with links back to this one to delve into the less glamorous, nitty-gritty details.  So stay tuned for some very focused articles detailing specific workflow obstacles in the next few weeks.  And then NAB is only a month away, which I am sure will bring a whole variety of new things to talk about.  I will be at Cineform’s booth as a demo artist/workflow consultant again this year, so feel free to stop by and check it out.

I am also involved in a shoot this weekend using both the Red One and the Sony EX1 for the first time, so that should provide some good info to write about.  So stay tuned, as I plan to resume posting regularly pretty soon.

3Gb SDI has very little hardware available to support it at this point.  The only real solution on the market at this point is to use Blackmagic’s Multibridge to generate it, and Blackmagic’s HDLink Pro to receive it.  This solution merely converts it DL-DVI, which we will discuss below.  The Multibridge allows fullscreen 2K without depending on the workstation video card.  In the future, I expect that this standard will be widely adopted to replace dual-link SDI.  This is especially beneficial as we move towards more 4K finishes, which currently require 8 HD-SDI links to support full resolution in realtime.

I am not aware of any solutions allowing 2K to be displayed over VGA besides using software to output the signal from a computer graphics card.  Certain older 21-22″ CRT monitors support 2K resolution, but you will need to be able to configure your software and/or your graphics driver to allow you to get fullscreen video output to the monitor.

Dual Link DVI can support 2K, but flat panel LCDs are the only products I am aware of that support this connection interface.  This makes it less than ideal for color grading work, but a great solution for visual effects work.  The excess resolution removes the need for dedicated full screen output, but that can usually be achieved as well.  Both computer graphics cards and the HDLink Pro can output 2K resolution over DL-DVI.  Graphics cards are obviously cheaper, and will likely work better in AE, but an HDLink Pro, with the appropriate hardware to drive it, will probably offer better performance in an NLE, like Premiere Pro.  It really depends on your budget, but if you can afford dual desktop monitors PLUS a full screen monitor, the HDLink Pro enables this configuration.

So those three interfaces dictate most of your actual display options in that regard.  Current 2K projectors are driven by Dual Link SDI, but we will see DL-DVI and 3Gb SDI replacing that in the future.  2K CRT monitors can be driven via VGA connections, and LCD flat panels can support 2K and above using DL-DVI.

The cheapest full resolution solution for viewing HD footage is definitely to use computer LCD monitors, usually 23 or 24 inch models, that have a native resolution of 1920×1200.  This allows 1080p content to be viewed pixel for pixel, but LCDs are inherently progressive, so issues based on interlacing will not always be visible on them.  Consumer LCD displays can be connected via DVI or HDMI, either directly to a computer graphics card, or through a hardware video I/O card, like DVI on an RT.X2 or HDMI on a BMD Intensity card.  DVI signal can also be generated quite efficiently from professional HD-SDI signals, so through the use of an SDI-to-DVI convertor, LCD screens can monitor HD video content without a computer involved.  There are also professional level LCD video monitors that accept SDI signals directly.  Some of the more expensive options can use this to display the full dynamic range available with 10bits per color channel, that would be reduced if converted to an 8bit DVI connection.  The new Sony BVM-L monitor is also supposed to correctly compensate for the display of interlaced content.  Cine-tal, eCinema, Panasonic, and JVC all have professional LCD monitoring solutions that are more affordable than Sony’s new top of the line LCD monitor.

CRTs used to be the gold standard for HD monitoring, but they are becoming harder to find.  They have always been very expensive, but they usually have a much higher contrast ratio than other technologies, due to their deeper blacks.  CRTs have a number of disadvantages, including low power efficiency, susceptibility to magnetic interference, and offer arguably unhealthy levels of electromagnetic radiation to viewers.  Sony has stopped making most CRTs, so people are scrambling to get them through second hand channels before they all disappear.  I for one can see the contrast difference, but appreciate the benefits of newer flat panel technology.

Projectors are becoming more popular for viewing HD material as the prices drop, and new technologies raise the picture quality.  Color grading certified projectors can be even more expensive than the older CRT solutions, but will provide a much larger viewing area, to better simulate the feel of a movie theater experience.  The brightness, color fidelity, and dynamic range are not as good in all but the most expensive products (>$50K), but the resolution and clarity is very high, even in the cheaper solutions. (<$5K).  The varying technologies powering different projectors provide vastly different results, depending on the price level.  Both DLP and LCD projection can have single imagers, using a color wheel or bayer pattern to achieve full color output, or three imaging chips, with one dedicated to each RGB color channel.  Many of the cheap products on the market that advertise 1080p resolution, accomplish this with a single imager, limiting color accuracy.  LCOS projectors always have three imagers, and currently offer some of the best color reproduction in their price range, at the expense of brightness.  Projectors scale all the way to 4K resolution with Sony’s SXRD line, for those with unlimited budgets.

There are also many consumer solutions that can be used to monitor HD footage, including Plasma Panels, rear projection DLP and LCD TVs, and many of them offer 1080p resolution.  The biggest disadvantage of many of these options is the lack of calibration options, as well as support for professional digital inputs.  HDMI has been changing that recently.  The new deep color standard for HDMI 1.3 will push consumer displays past the 8bit barrier, and open up many more affordable options for high quality HD monitoring.

 Until then, my reccommendation would be to use an LCD for most editing work, and only look for a more expensive solution for the shorter period of time that you spend color correcting near the end of the post-production process.  If you really want to dive off the deep end, you can explore solutions that offer 2K or higher resolution, but I don’t intend to go beyond my 30″ LCD in that department, 8bit DVI limit or not.

As newer post-production software is developed to squeeze every last bit of available performance out of available hardware, the GPU is becoming a more important factor in building a high performance workstation.  A few pieces of software that I use that depend on the GPU are: Matrox’s AXIO-LE, Red Giant’s Magic Bullet (especially Colorista), and Iridas’ SpeedGradeHD.  Each has a list of supported cards, and hopefully there will be some intersection in those subsets, or these software applications will be incompatible with each other.  For any given product, there are usually a variety of options, sometime ranging in price from $50-$2500.  Determining which of these options best suits your needs is an important decision, and sometimes the best choice is not immediately apparent.

The competition between NVidia and ATI used to be much stronger, but recently, NVidia has pulled ahead significantly.  I am not sure if this is related to ATI’s abrupt acquisition by AMD last year, or anything else, but NVidia’s development has been consistently resulting in products that are much more capable than ATI’s.  In the professional arena, ATI doesn’t even offer features like SDI outputs and Genlock, to compete with NVidia’s offerings.  These specific features are very relevant to the utilization of these cards in the post-production workflow.  SLI is another NVidia development that ATI has no answer for in their professional line, but implementations of that technology are more tailored to 3D animation and scientific applications.  Stereoscopic output has been offered by NVidia’s QuadroFX line for many years, although their solution is a bit outdated at this point.

ATI has few advantages to counter with.  The most significant one I am aware of, for post-production, is that the ATI architecture is better optimized for returning processed images to the system bus.  Certain applications are able to pass more data to and from ATI cards than their Nvidia counterparts, which is beneficial if you plan to do more than preview the results on screen.  This is why Matrox’s AXIO-LE gets better performance when paired with ATI cards than much more powerful NVidia solutions.  Another issue I have seen with Cineform’s RT engine in Premiere is a color shift between between the video overlay and still frames.  According to David Newman at Cineform, this is due to an inconsistent implementation of YUV overlay on NVidia cards (See his comment on the ProspectHD post) and ATI cards, to their credit, do not suffer from this problem.  There are very few other features in ATI’s favor that I am aware of, but I am always open to being enlightened in that regard if I am overlooking something significant.  Given the current state of things, my recommended choice in most cases, would be to go with an NVidia based card.

Choosing between NVidia and ATI solutions is not the only significant step in the selection process.  Frequently, the most confusing aspect of choosing a new display card, is motivated by chipmakers’ desire to make higher profits from business customers, in that “professional” 3D graphics cards are much higher priced, than seemingly identical consumer gaming 3D graphics cards.  The actual specific differences are rather vague in many cases, and will depend on the requirements of your application.  Certain features such as SDI output and Genlock are clearly exclusive to professional hardware, and product support is much better for the professional lines, but when it comes to GPU processing, the differences are not so obvious.  This is especially true since both companies utilize a unified driver architecture, allowing the same drivers to support almost any of their cards.  Both companies throw around the term OpenGL in regards to their professional cards, but most of the same features are available from the consumer cards.  I have used OpenGL acceleration in After Effects, and have found no real differences, but I am not a professional animator, so higher end 3D animation and modeling programs might see certain advantages.

ATI has their FireGL line of professional cards to compare to their Radeon series.  I have used very few of these cards, so I can offer little in the way of advice.  They are rarely recommended or required by post-production software solutions.  My primary experience with the Radeon line has been in conjunction with the Matrox AXIO-LE, and I have not been impressed with the stability or features of the cards.  The most important feature that I find totally unsupported is the hardware spanning of two displays.  I also have occasional vertical sync issues when running LCDs at 1920×1200, but all this is based on my experience with two X1900 series cards.  I have much more experience, and a greater level of success with NVidia cards.

Nvidia’s QuadroFX line of professional graphics cards is VERY similar to their GeForce line of cards, and with even greater price differences.  In my experience, most software runs equally well if not better on GeForce cards compared to their QuadroFX relatives.  I own a QuadroFX3400 which is almost exactly identical to the GeForce6800GTX, and was four times the MSRP when I bought it.  Although the card has served me well, I have found no compelling reason to have required it over the similar GeForce option.  There is a rumor that Nvidia disabled certain functions when they released their newest generation of consumer cards, that will now only be available from the QuadroFX line, but I have not been able to confirm that.  Specifically they are said to have disabled hardware support for full screen video overlay, (allowing full screen preview in an NLE) which I intend to test once I get a working GeForce8 card.  I would appreciate information about anyone else’s experiences in this regard.  If that is true, it means that we might soon find signicant disadvantages from using consumer cards for professional work, but fortunately, I do not think we have yet come to that point.

What all that boils down to is, currently Nvidia is the performance leader, and unless you have a compelling reason to shell out the money for a QuadroFX model, a GeForce card should be suitable for most applications.  That said,the new 8800GT is a remarkable value for almost anyone who needs a powerful GPU. (Please note I am NOT speaking of the much lower end 8600GT card)  As an added benefit, the new 8800GT should run cooler and quieter than any other card with similar performance.  I also expect that the new PCIe 2.0 compatibility should be able to be taken advantage of with upcoming release of the next generation of Intel Xeon workstation platform early next month.  If I hadn’t been in the process of acquiring the similar 8800GTX, which is at least twice the size, price, heat, power, and noise, for similar resulting performance, I would have already ordered a GT by now, and still might do so regardless.

The simplest solution that provides HD-SDI input to a laptop is the Motu V3HD.  Connected via Firewire, it allows capture of digital and analog High-Definition video signals at DVCPro-HD quality.  Limited to 1280 pixels in width, and 100Mb/s, this is a lower end HD solution, but bears mention none the less.  I have not personally used one, but it is supposed to be compatible with Premiere Pro CS3, as well as Final Cut Pro.  The data rate and processing requirements allow this format to be used on most high end consumer laptops, but those looking for full resolution 1920×1080 solutions must look farther.

The next solution is currently only available to Mac users in Final Cut Pro, but is a significant  technological development.  AJA’s “I/O HD” is a Firewire800 based solution that can capture and playback full resolution material, with 10bit color, in Apple’s new ProRES codec.  Although not a PC based solution, it does enable mobile users to capture high quality, full resolution footage.

Anything beyond that will involve a bit of creativity, and what follows is highly speculative.  Newer laptops have replaced PCMCIA card slots with ExpressCard slots.  The new formfactor is much simpler, and has two basicinternal variations.  The slot has pins available to interface directly into the USB subsystem (480Mb/s) or directly into the Southbridge via the PCIe x1 interface (2000Mb/s).  The PCIe interface provides an ExpressCard slot with enough bandwidth to support uncompressed HD video, at least 10bit 422 at 1080i/p.  RGB 444 might even be possible at 24fps, but that would depend on how much overhead was imposed by the interface itself, among other things.  This bandwidth has been utilized in the design of the ExpressCard option for the CalDigit HDPro, but having a single slot with the capability of transfering video at uncompressed data rates leaves us with a problem.  If the ExpressCard slot is being used to connect some form of video I/O interface, how do I connect my storage at uncompressed speeds.  Unless you find a laptop with two ExpressCard slots, you will not be able to use both at once.  I guarantee that the capture card is necessary for realtime full resolution HD acquisition, so how can we do it without using the high speed storage?  Compressing the video becomes the obvious solution.  So a capture solutionis needed that allows realtime compression, and can be jury-rigged to connect to an ExpressCard slot at PCIe x1 bandwidth.

A company named Magma has developed a solution that really opens up the available options.  Their ExpressBox Pro product allows a PCIe card to be inserted and connected to a laptop via an ExpressCard slot.  At the very least, PCIe x1 cards can be expected to work, and ideally higher end PCIe x4 based capture cards may function properly as well.  After all, the HD video data itself is usually well under 200MB/s, depending on the specific settings and format.

The first option that comes to mind are the Intensity cards from BlackMagic Design.  They allows full resolution capture of 1080i/p at up to 10bit color in the 422 YUV colorspace, over HDMI or analog on the Intensity Pro.  Convienently, Blackmagic also makes an HD-SDI to HDMI converter, the HDLink, so we can use this to pump HD-SDI into the Intensity card.  Blackmagic also has a MotionJPEG codec that we can capture directly into, so it would seem that they offer a fairly complete solution to our problem.

Another option using the same hardware is to use Cineform compression, as detailed here.  In my experience Cineform’s compression results in a higher quality final picture than Blackmagic’s current implementation of MotionJPEG codec.  The downside of using Cineform is that they don’t support live playback, out of the Intensity card the way Blackmagic’s codecs do. If you have an external monitoring device available, this can be a very helpful option when trying to edit on a small laptop screen.  To Cineform’s credit, they allow you to use the secondary display output from your laptop as a full screen video output if your graphics card supports it.

Our next PCIe x1 based solution is the RT.X2 from Matrox.  Although I have not been able to confirm that this has ever been successfully used in this capacity, it remains a theoretical possibility.  The RT.X2 would be advantagous in that it would offload much of the compression processing from the laptop CPU to the PCIe card.  It would allow analog HD capture, but would be limited to 1440 horizontal resolution, and would allow preview via DVI or analog HD.  On the positive side, with hardware acceleration, Matrox’s MPEG I-Frame codec would probably give the best creative editing performance of any of the solutions we are examining here.  Realtime effects and exporting would be advantagous for the editing process, but the original footage acquired would not be as high quality to begin with.

In theory, the Magma ExpressBox could support other cards.  While the bandwidth is limited to the 2000Mb/s (200-250MB/s) of the ExpressCard’s PCIe x1 bandwidth, the phyical connector in the box is a PCIe x16 slot.  It would be interesting to know if it would support an AJA LHe or a Decklink HDPro.  The AJA card would allow 10bit capture into the Cineform ProspectHD codec at full resolution, and Decklink might allow RGB 444 capture at 24fps.

There is one more Blackmagic based option that I know isn’t fully developed yet, but seems very close.  The Blackmagic Multibridge is based on the same technology that allows the Magma ExpressBox to work, external PCIe.  If an ExpressCard could be fabricated that interfaced the ExpressCard PCIe x1 bus to the DVI shaped cable that the Multibridge uses, that would be a great solution.  When the first Multibridge Extreme was released, it was listed to be compatible with PCIe x1 slots, at least at SD resolutions.  PCIe x1 has the bandwidth for HD if used efficiently, and the Multibridge has many I/O options, so I think it would be the ideal portable solution.  I know it can capture to MotionJPEG, and I believe Cineform includes capabilities to capture from it into their codec as well.  I have not been able to confirm that, but it is implied on their website.

The last option I will mention has been discussed and rumored about for years, but I have yet to see a product hit the market.  Why not have an ExpressCard with HD-SDI I/O directly on it?  Heat will be an issue that needs to be overcome, and mini-BNC connector could be used to improve the form factor of the physical connections.  Ideally if it was a Blackmagic product, it would support live capture into MotionJPEG, Cineform, and ProRES on a Mac, for maximum possible market.  If/When it gets developed, I know it will sell well, assuming it functions correctly in an established normal workflow.  Realtime compression will be required for any laptop solution, but this doesn’t have to be accomplised in the card itself, it just has to be compatible with it being done by the CPU.  I look forward to seeing a product like this released, as it would greatly enhance the workflow for portable post-production solutions.

Technology has come a long way in the last two years, especially in the CPU processing aspect of the equation.  A well equipt laptop can be purchased now that has more processing power than the highest-end Windows based workstations of two years ago, thanks to the Core2 Duo.  I bought a Xeon workstation in 2005, and one year later, bought a 12″ notebook for LESS money, that has MORE CPU power.  With the upcoming release of quad core mobile CPUs, we can remove processing power from the list of limitations that mobility imposes.

Next is RAM, and we are in a unique situation in that regard.  Most systems still use 32bit OSes, and are limited to 4GB of RAM.  This software limitation has allowed notebooks to catchup with desktops in this regard, as demand has not climbed as much past 4GB in the desktop sector, and notebooks were under no similar limit until they caught up.  4GB of notebook RAM can be had for under $200.  Obviously mobile solutions will not be limited by the maximum available RAM. (Any more than a desktop)

The first area where we encounter trouble is with storage, in both capacity and transfer rate.  While there are solutions that allow uncompressed HD speeds and capacities on a laptop (ExpressCard to CalDigit HDPro), that solution is not very portable.  The capacity issue can be solved via 1TB drives connected via firewire, or internal RAIDs or 2.5inch hard disks in large laptops, but there is currently no way to provide the required transfer rates for realtime uncompressed high-resolution editing, in a mobile solution.  This leads to a need for a different solution.  We can utilize the extra CPU now available by using a compressed video format to decrease the strorage requirements.  (As an aside, my FREE IDEA of the day is: A mobile array of 8x250GB 2.5 inch disks with an external PCIe interface for use with x1 PCIe or ExpressCards would offer 2TB at 250MB/s.  It would be an interesting solution, and if a company decided to create it, all the technology already exists.)

 Since there are no reasonable storage solutions for uncompressed, we must examine compression options.  The first standardized option is HDV.  The advantages are low-bitrate, wide support, and firewire I/O, which most laptops already have available.  The disadvantages are lower quality, 8bit 4:2:0 MPEG encoding, and a 1440 horizontal resolution limit.  The next option is a very recent one for Premiere Pro users, with DVCPro-HD, made possible by the release of the 3.1.0 update this week.  With datarates of 5-12Mb/s, this format is will within the 30MB/s capabilities of a single 2.5inch laptop drive.  The horizontal resolution is even more limited, to 1280 pixels wide, but it encodes 8bit 4:2:2 with DCT compression, and in general a higher bitrate should improve quality.  Since it doesn’t use MPEG compression, it should be less CPU intensive to playback and edit, leading to better performance.  The next step would probably be one of Cineform’s products.  AspectHD is limited to 1440 at 8bit, but it still the highest quality solution yet, usually at around 9-10MB/s, using Wavelet compression, which allows efficient low resolution playback as an added bonus. 

 Blackmagic Design released a MotionJPEG codec a while back that allows full 1920×1080 files at 8bit 4:2:2 to be used with a data rate of around 12MB/s.  The advantages are that it can be used for free, with out any limitations that I am aware of, but it is designed to be used with their I/O hardware for acquisition and preview.  The only disadvantage is not really a disadvantage comparedto the options below it, but image quality will not be as good as Cineform, and is limited to 8-bit.  Cineform’s higher-end product ProspectHD has few limitations, allowing 10bit 422 at full1920x1080 to be edited at around 15MB/s depending on settings.  This is easily sustainable on internal laptop disks, while an external firewire drive could increase performance and capacity.  Currently Cineform is the solution I would recommend if you need high end HD editing in a portable form factor.  The next question that Cinefrom prompts is, how high can I go, and currently 2K 444 RGB is possible at 30-40MB/s, meaning an internal RAID 0 would benefit playback on a laptop.  There is even talk of 4K realtime playback with the release of the RedOne, so it seems that the sky is the limit.

Other solutions that I am aware of, but don’t seem ideal are: Matrox MPEG I-Frame files from AXIO or RT.X2 at 12MB/s with the Matrox M.key in desktop mode, which I use, but the performance is not good for creative work without the hardware acceleration.  In the non-Adobe world, FCP offers DVCProHD, and now recently ProRES, a full frame 10bit 422 codec that runs about 15MB/s at 1080p I believe.  I have no familiarity with Avid, but XpressPro or Media Composer with DNxHD might be a portable HD option as well.

Laptops have done a lot of catching up recently, and the concept of ”desktop-replacement” is a much more legitimate now than it was two years ago.  The most important aspect we have not yet examined is HD I/O for portable solutions, especially for portable acquisition, which I plan to go over next time.

Let’s start with SCSI, since it is the easiest to dismiss.  While we are discussing the connection of SATA drives, many of the first generation SATA arrays had intergrated controllers and Raid hardware, and then needed a fast connection to the host.  These arrays were designed to replace much more expensive SCSI drive based arrays, so the target customers trusted the SCSI interface, and already had high end SCSI controllers in their systems.  That made SCSI the optimal connection solution for early SATA arrays.  The SATA Raid controller masquerades the entire array as a single SCSI disk, allowing connection to systems through existing SCSI cards.  With up to 320MB/s of bandwidth, a single SCSI channel can efficiently support 5-7 SATA disks without much impact on performance.  The biggest reason to dismiss SCSI as a serious possibility is that eSATA is a better option for most, and the remaining will be much better served by a Fibre Channel interface, allowing for economical upgrading to a full SAN in the future.

The next step for high end SATA arrays was to replace the SCSI emulation with a much more flexible interface, Fibre Channel.  With up to 400MB/s, Fibre Channel has few disadvantages to SCSI, and one major benefit.  SATA disk arrays with Fibre Channel interfaces can usually be connected to switches, and shared between multiple systems, in a SAN.  All connected systems get direct block level access to the disks, which will almost always be faster and more responsive than sharing through an ethernet network.  With the proper Shared SAN software, these systems can also share the data down to the level of individual files.  For facilities where multiple users do collaberative work, based on the same source data, Fibre Channel is probably worth the added initial investment, even if a SAN is not immediately implemented with the purchased hardware.  The possible extensible use of an array beyond a single workstation should be well worth the increase in price, and as an added benefit, cable lengths can easily be increased enough to keep the noisy array out of what should be a peaceful creative environment.

 There are many products available that share storage directly to an ethernet network connection.  The consumer varients hardly have the performance to support DV editing, let alone anything more demanding.  The higher end options, with prices similar to SCSI and FC do offer some interesting possibilities, but will rarely be the optimal choice for a given situation.  Any gigabit ethernet connection is limited to 125MB/s, and in reality, the achievable performance is usually about half of that.  Gigabit network solutions will not be a solution for uncompressed work at HD or higher resolutions.  10Gb Ethernet would offer the desired performance, but is not currently an economical solution.  If compressed files are used, regualr gigabit ethernet can be used to transport the data in realtime, but I would still argue that arrays interfacing directly to ethernet are not the most efficent solution.  Any similar array directly connected to a workstation through a different interface will give much better performance to that system, and can still be shared on an ethernet network via that workstation.  There will be a performance hit on that station when sharing data to other system, but a network card with a TCP/IP Offload Engine (ToE) can minimize that effect, and the increased performance on that system do to the high speed storage directly attached should more than offset whatever is remaining.  This would involve using an array with one of the other interfaces we are examining.

A recent technology that uses ethernet to transfor data, is iSCSI.  Promoted as having many of the advantages of Fibre Channel SANs, iSCSI gives initiator devices (workstations) block level access to their target device (arrays).  This allows the target device on the network to emulate a local device on the initiator’s system.  The downsides are that maintaining data intergrity on shared target drives, requires most of the same expensive software infrastructure that a Fibre SAN does, and the inefficiencies of the TCP/IP protocol are still present to limit the realistically achievable maximum transfer rate.  If you have to deliver identical data to a large number of systems, and don’t want to spend money on the performance that Fibre Channel hardware can deliver, then iSCSI might be of benefit to you.  These products are targeted at large corporations, and don’t scale down in size without losing performance, and maintaining deployment complexity.  I don’t see this being the solution of choice for most desktop PC workstation professionals in post-production field.

 The next solution is offered in a staggering varietly different solutions, eSATA.  This can be fairly confusing due to the number of variations of this technology on the market.  eSATA is a very flexible standard, but not all implementations will deliver optimal results.  For example, some products support port multiplying to increase the number of drives without increasing the complexity of the interface cables or the Raid controller.  This solution is good for high volume solutions, but will not deliver the same level of performance as direct connection based solutions.  The simplest, professional level, eSATA array will be an external drive enclosure that passes each drive’s data interface directly back to the controller, which will usually be some varient of PCI card, inside the workstation.  This gives the card direct full-speed access to each disk drive, and all Raid processing is done on the controller card inside the workstation.  This will be the fastest and most efficient solution for the cheapest price, and I highly recommend it.  The limitations are the cables which usually have a 6 foot maximum length, and the fact that Fibre channel is easier to share.  But for the independent, budget conscious, single workstation user, this is the way to go.  Eight disks gives you enough storage for almost any concievable independent project, and eight drives should support uncompressed HD if desired, and may even work for 2K with an efficient Raid controller.  Solutions that use port multipliers to connect more drives, will increase storage but not performance, and usually require more expensive SAS compatible controller cards to support the port multiplying.  If you need more than 8TB of storage on your system, these might work well for you.

The most recent development in this area is the advent of the use of External PCI Express as an array interface.  A small PCIe passthru card is all that is required in the host system.  An x4 slot can transmit and recieve 10Gb/s of data, which is 1.2GB/s, and there is much less overhead than most other interfaces.  An x8 slot is capable of twice as much throughput for an insignificant margin cost increase.  With External PCIe, the drive controller and raid processing electronics are contained within the drive enclosure, and the controller has direct access to the disks.  As a result, the array could easily be moved to another system, without having to bring a separate controller card from within the system.  Each system would need an External PCIe bracket, but those are only forth about ten dollars.  Due to the nature of the External PCIe interface, the computer has the same level of access to the controller and its data that it would if those electronics resided on a board contained within the workstation.  Another benefit of PCIe, is that the new ExpressCard for notebooks is based on the same interface.  This allows a simple adapter to connect an External PCIe device to a notebook at x1 speeds (over 250MB/s will be fast enough for uncompressed HD).  Currently I am only aware of two vendors offering soluitions using this technology, CalDigit and Ciproco.  It will be interesting to watch as this technology continues to develop.

So my recommendation is that high end eSATA solutions are the most economical direct attached storage solutions, and can support uncompressed HD if needed.  Larger operations that are considering upgrading to a full shared SAN system in the future will probably find the increased initial investment of Fibre Channel arrays to be well worth the value when they re-utilize the same hardware in their SAN implementation sometime in the future.

The industry has responded with many different solutions, that vary in concept beyond recognition and in price by many orders of magnitude.  The earliest solutions involved video tape, analog replaced by digital recording.  Hard disks were introduced for random access to data, and now those are slowly beginning to be replaced by solid state flash chips.  Since this site is targeted to PC users, we will focus on hard disk based solutions, and the interfaces with which they can be accessed by a media workstation.

Hard disks are produced with five popular interfaces:  IDE/ATAPI, Serial-ATA (SATA), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), and Fibre Channel (FC).  IDE and SCSI interfaces are currently being phased out and replaced by their more capable and flexible Serial varients.  I know little of true Fibre Channel hard disks, but that format is rarely used in this industry.  That leaves only two options, which are now somewhat similar and compatible, SATA and SAS.  With identical connection cables, and both offered in 3.5″ and 2.5″ form factors, it is hard to tell the two options apart visually.  Their interfaces both support 300MB/s, dedicated buses for each drive, and port splitting when that is not required.

The biggest differences between SATA and SAS are performance and cost, which eventually distill down to one issue: size.  SAS disks have slightly more capable and efficient electronics, run fewer platter, with less data, and much higher RPMs and faster I/O and transfer rates.  SATA drives usually have much more storage capacity, lower speeds, and are always much cheaper.  At first glance, high end post production work would seem suited for SAS drives, since moving picture footage requires a higher data transfer rate than almost any other application of computing technology.

There are four other factors, which when combined, weight much more heavily in favor of SATA.  The first is price.  Since the difference in price per Gigabyte is currently so great, and SATA drives are not that different in their design or performance, a few quick calculations will reveal that while SAS disks have higher performance per drive, SATA disks deliver more performance per dollar, regardless of their storage capacity.  Second is that the infrastructure needed to aggregate the performance of multiple disks (Raid arrays) will be required, regardless of which disk solution we choose.  This is due to the fact that HD resolutions and larger require much higher data transfer rates than any single drive can provide (unless compressed, and even then, fast disk access is beneficial).  The marginal cost to increase the number of drives being aggregated will be low in many cases.  The third factor is that digitalized footage requires a tremendous amount of storage space, once again contributing to the need for many hard drives to be combined.  Lastly, most of the popular solutions to improve reliability, do so by utilizing even more capacity, to store redundant information in the form of parity, or straight backups.

These factors, when combined make a strong case for SATA disks, which have higher capacity at the expense of performance per drive.  If we are combining drives anyway, the performance benefits of SAS will usually be negated by combining more SATA drives for less money.  This is a case where quantity can clearly overcome quality in most instances.  As a side benefit, SATA drives usually have much greater capacities.

The only time when SAS may be favorable, will be when there is little need for high capacity, and when there is value to smaller solutions.  Fewer SAS disks are required to reach a given level of performance, and will therefore be more portable, require less power, and frequently generate less heat and noise.  For visual effects, were a few seconds of footage are manipulated at very high quality, or short commercials, SAS may be a more efficient option.

In most cases though, the numbers come down in favor of SATA by along shot.  Let’s imagine a two hour movie, with a 10:1 shooting ratio, giving us 20 hours of footage, and for the sake of example, let’s assume a data rate of 100MB/s.  With 3600 seconds in an hour, that is 360000MB an hour, or 360GB.  20 hours of footage would require 7.2TB of storage.  Add 10% to avoid disk fragmentation, and you need an 8TB array.  With 1TB SATA disks you need 8, plus two more to support Raid 50.  You will have the bandwidth of eight drives, and assuming 50MB/s each for SATA disks, and an efficient controller interface, that is 400 MB/s, more than enough for our 100MB/s files.  10 SATA drives at 1TB currently costs ago $3,000, and the Raid hardware will be required by both SATA and SAS, so it does not necessarily need to be factored in.  Now when onlining a production, not all footage is usually captured, but when you factor in captures, conformed exports, film and video colored versions, testless and texted masters, a 10:1 ratio will not be an inaccurate estimate.  Now I used round numbers, so that if the datarate of your format of choice is higher or lower, you can ajust accordingly. 200MB/s footage would need 20 disks, but could get double performance.  50MB/s footage would only need 5 disks, but could still expect 200MB/s of performance.  Have less footage, I left a 4x overhead in this example with 20 hours of source, but I also used 1TB drives for my calculations.  With 10 hours, 500GB drives show SATA to be even more economically favorable.

Now for a quick comparison to SAS, we start by noting that the maximum capacity is 300GB, and you can expect to pay at least $500 per disk.  Our 8TB example would require about thirty disks, assuming a Raid 50, striping together three Raid 5 arrays of ten disks. 27 data disks is 8.1TB for a cost of $15,000 in drives alone, not counting that it requires hardware for three 10 bay array enclosures instead of one.  From a performance perspective, assuming 80MB/s per disk, you can get over 2GB/s if you want to pay for an interface that fast, but remember that this is all for footage that is 100MB/s.  2GB/s might be good if you want to share it between multiple systems, but with that many users, usually multiple productions will be processed concurrently, requiring much more storage capacity anyway.  By multiplying up and down for different formats, it becomes clear that there is no way that SAS can economically catch up.

So I hope this successfully establishes that SATA disk drives will almost always be the drive type of choice for post-production environments.  I plan to examine the different options for connecting these drive arrays to a workstation or group of systems in my next post.