Speaking of DSLR files, Adobe has totally reinvented the way they are handled in CS5.  Most applications, including the CS3 and CS4 versions of both Premiere and After Effects, use QuickTime importers to access the content of Canon DSLR files.  This makes sense, since they are stored in an MOV wrapper, but leads to two issues.  One is that is specific to Adobe is that on a PC, QuickTime files go through a few extra steps before they are accessed by the application, so there is a performance hit, and with lots of files accessed at once, there are usually stability issues as well.  The other issue effects all applications that use QuickTime to access DSLR files, and that is that ever since QuickTime 7.6.2 was released, Canon DSLR files have been decoded in a much more flat and washed out color space than they were designed to be viewed in.  Prior to version 7.6.2, they were decoded in a way that clipped the highlights and shadows, which was even worse.  In CS5, Adobe worked with Mainconcept to create an importer that reads the DSLR source files without involving QuickTime at all.  This alleviates both the performance hit on PC systems, and the color space issues of QuickTime’s default decoding.  A lot of work was put into getting the decode matrix and color space exactly correct based on the processing that Canon’s hardware does to the file in the camera.  This should allow CS5 to decode the files more correctly than any other application that I am aware of, and give more options for color processing at later stages in the workflow pipeline, since more of the original color data is preserved.

The Mercury playback engine has a significant impact on the user experience, with most frequently used tools being available in real time.  Supposedly most of the decode and playback improvements are based on the new code written for native 64bit execution, with the GPU offload limited to effects processing.  While many editors don’t use very many discrete effects in their work, there are some intrinsic playback functions like scaling frame sizes and adapting frame rates that are considered effects and offloaded to the GPU.  This allows content of different frame rates and resolutions to be intercut seamlessly on the timeline, and it is truly seamless.

I have occasionally found myself editing in the wrong sequence frame rate without even noticing it, since the software makes the conversion on the fly.  Even more frequently I have found a clip shot in the wrong frame rate on a tapeless camera almost escaping detection because gone are the red render bars and playback glitches that used to stand out.  This allows editors to import media from many different sources without prerendering everything to a normalized format.  I used to spend about a quarter of my time at work converting strange source footage into 1080p24 intermediate files, because any footage not matching the timeline format would cause previous versions of Premiere to glitch during playback and occasionally crash.  Now I would recommend carefully converting any footage used in a final piece to the correct format for maximum control, but this step can now be put off until the online stage, since it is no longer required for playback and stability.  Since 90% of footage usually ends up on the cutting room floor, putting off these time consuming conversions until after the creative edit is finished, will drastically reduce the amount of footage you end up needing to process.

So this clear increase in performance leads to the question of: how far can you push it?  A few months back I processed an ISO noise test in After Effects CS4 for Shane Hurlbut.  We were comparing the image noise produced at twelve different ISO levels on the Canon 5D, and the project involved twelve streams of video with masks, levels, and position adjustments for a tiled view.  I was getting about two frames per second when rendering previews, which seemed reasonable considering the amount of processing involved.  When I saw the list of GPU accelerated effects in Premiere Pro CS5, and was asked to create torture test for Adobe to show off at NAB, this jumped to mind.  I recreated the entire project, using twelve layers of native DSLR footage, each layer having a motion effect and a 4-point garbage matte to create the tiling, and a color correction applied to exaggerate the noise to a clearly viewable level.  The same basic setup that was getting 2fps in AE CS4 played back in real-time in Premiere Pro CS5.  (Dual Xeon X5365 CPUs and 16GB RAM with a QuadroFX 4800) Needless to say I was quite impressed with the outcome, since it was deliberately beyond Adobe’s ten-layer playback claim, and using a complex format to decode and playback.  Clearly GPU acceleration can have a dramatic impact on application performance.

There has been much discussion and debate on tech forums and blogs about the specifics of Premiere Pro’s hardware support for CUDA acceleration.  Adobe has severely restricted the number of cards for which they officially support CUDA based GPU acceleration, to maintain control over the hardware environments upon which their accelerated code is tested, supposedly for stability reasons.  The official list is limited to QuadroFX 3800, 4800, and 5800, as well of the discontinued GeForce 285 GTX, with certain limitations, for those on a lower budget.  There are currently no officially supported mobile GPUs, even though notebook CPUs are usually more in need of a performance boost than desktop chips.  This may be due to the fact that even the newest mobile QuadroFX 3800M is still based on the G92 core from the GeForce 8000 series, but I don’t like seeing software artificially limited in regards to performance or hardware support, and this is an example of both.  Don’t confuse legitimate limitations and artificial limitations, since clearly a powerful GPU is necessary for optimal performance in CS5, but there are cards of equal capability that are specifically excluded from the list, supposedly for stability reasons.  Luckily Adobe has left an option for knowledgeable users to override some of those artificial limitations, and I anticipate seeing them being dropped completely in a future update.  I anticipate a more reasonable requirement of any NVidia card supporting CUDA 1.1 or 1.3, with at least 785MB of video memory, at some point in the future.

While Premiere Pro CS5 is not perfect, it is a complete reversal from the previous fiasco that was CS4.  It clearly demonstrates the possibilities provided by GPU acceleration, resting solidly on 64bit code with proper multithreaded programming, it scales to take advantage of whatever hardware is made available for it.  Since Adobe has made a practice of introducing significant improvements in incremental dot releases, I am looking forward to seeing how else they refine it in the coming months.

FTC Disclosure: I have been on Adobe’s beta team for many years, and Adobe has provided me a copy of CS5 for this review.  NVidia has provided me with graphics hardware in the past, which I utilized in this review.  My only admitted personal bias is my preference of Windows over OSX, because I like full control over every aspect of my computing experience.  If for some reason that bothers anyone, there are plenty of other sources of information on the internet, but I try to provide unique insight on how each of these tools fits into the larger post-production picture.  Any relevant critique or response is welcomed.

The other major new feature for this release of After Effects is the Rotobrush.  Based on Photoshop image processing functionality, extended to account for the information made available from multiple frames, this is the closest thing you are going to find to magic in a current generation compositing application.  It allows you to automatically separate foreground and background objects in a video image, otherwise known as rotoscoping.  Of course it is not perfect, but it is a huge advance, compared to previous tools.  Automated tools like this are usually much more impressive if you aren’t expecting a miracle going into the process, but as long as you have reasonable expectations, Rotobrush can give you usable results in a rush, or a good starting point when more precise work is required.  Much of the “magic” of the new tool is in the processing of the automatically defined edges, and this capability is available without using the Rotobrush, in the form of the “Refine Matte” effect.  This effect can be used to polish the edges of standard keys, or manually rotoscoped footage.  Most of the other major feature enhancements for this release come in the form of integrated third party plug-ins, including Color Finesse for grading, Mocha for tracking, and DigiEffects Freeform for 3D simulation.  Each of these are feature rich plug-ins that include functionality that is beyond the scope of this overview, but are well worth experimenting with if you spend a lot of time working in After Effects.

There are quite a few new features in Premiere Pro CS5 that don’t hinge on the new Mercury Playback Engine.  These relate primarily to metadata and content organization, as opposed to the media itself.  Speech detection tools were introduced in CS4, and now those can be used in conjunction with the features of Adobe Story, to link your actual footage to your original plans, including scripts, storyboards, and shot lists.  Much of this footage processing links to OnLocation CS5′s features, which besides monitoring the technical details of a captured media signal, allow ingest of metadata during the acquisition process.  Building up as much information as possible during production, on a per-shot and per-take basis, will further streamline the organization process during editorial.  All of this information is searchable, with the intent being that an editor will have a much easier time finding what they are looking for, as this metadata is passed down throughout the post production workflow, based on source timecode.  This content logging metadata is not the only way that CS5′s handling of non-media information has improved.

Adobe has also been working hard to streamline the process of exchanging existing projects with Premiere from Final Cut Pro and Avid Media Composer.  They have further refined the FCP XML import and export functionality that was first introduced in version 4.0.1, which offers some interesting options for Mac based workflows.  Working at a PC based facility, I have had more opportunity to take advantage of Premiere’s support for exchanging timelines with Avid.  My current workflow is to use EDLs to move my sequences from Avid to Premiere, but with CS5′s improved support for AAF import and export, hopefully I will soon be able to leverage the additional functionality offered by AAF exchange.  (One caveat with using EDLs to move sequences, is that occasionally Avid adds spaces to the end of every line, and CS5 will not relink to the source footage automatically, unless you manually remove those spaces)  I have yet to find the optimal settings to switch to AAF files, but I have talked with people who have it up and running.  Combining this sequences exchanging feature with CS5′s support for DNxHD in both MXF and MOV wrappers, offers some interesting possibilities, but the truly revolutionary workflow will finally arrive when Media Composer 5 is released next month, allowing Avid to link to external QuickTime files thru AMA.  Hopefully this will allow Premiere Pro CS5 and Media Composer 5 to exchange sequences that all reference the same source files, without any conversions or transcoding.  Since Canon DSLR files will be supported natively both applications, this should totally streamline my current workflow.

Now why would someone want to move their project between different NLE applications?  An editor’s familiarity with a specific toolset is usually important to them, so certain offline creative editors are not going to switch their primary editorial app no matter what.  To its credit, Avid has a solid reputation for handling extremely large projects with no decrease in performance.  While Premiere Pro has taken a huge step forward in that regard, it is going to take a long time for it to build a stable reputation, since trust usually develops very slowly.  On the other hand, Premiere makes a perfect interactive online conform tool, especially compared to Avid.  It can ingest most digital formats in their native form, supports SDI capture and playback over a number of different hardware solutions, and scales to 2K and 4K resolutions, in RGB color space if desired.  It is compatible with DPXs for color grading, and can playback surround sound for reviewing final mixes.  The fact that Premiere can use Dynamic Link to ingest your visual effects changes from After Effects, and to output to Encore for adding interaction becomes an added bonus for this conform solution.

Dynamic Link is now on its fourth iteration, and has matured into a functional tool, even at HD resolutions.  It is designed exchange media between apps, without having to waste time or disk space by rendering.  While it started as a method for importing AE comps into PPro, it became the backbone of PPro’s export capabilities thru Adobe Media Encoder as well.  While Media Encoder CS5 is a dramatic improvement from the previous version, Adobe also added back in the option to export files directly from within Premiere.  This is much quicker for rendering out small parts of large projects, avoiding the overhead of syncing the entire project with Media Encoder before rendering a couple of frames.  They also brought back the ability to easily export single frames from the timeline, a function that had been removed in CS4, an inexcusable oversight that has now been rectified.  Exporting a Premiere Pro sequence thru Media Encoder, while not transparent, is now a much faster and more stabile process than it was in CS4.  Media Encoder also now supports DPX sequences and AVC-Intra MXF files, for both input and output.  Still sequence support in both Premiere Pro and Media Encoder is still missing crucial options, for relinking footage in PPro and manually overriding the frame rate in AME.  Hopefully we will have more user control of these settings in future versions.

One thing that I would like to see handled differently in Media Encoder, as well as the Premiere exporter, is the default settings for each export plug-in.  Adobe is never going to be able to predict what settings every user is going to want, but it would be nice if it defaulted to whatever I selected last time I used that particular output option.  As it stands, Media Encoder defaults to the same output as the last file queued, but if I choose any other output (AVI, MOV, DPX) it returns all of the individual settings to the original default, which is usually DV based.  Now it is possible to save presets, but I hesitate to do so unless I anticipate using that exact configuration frequently, because sorting through too many presets can become harder than manually defining the individual settings.  On a relate note, be careful where you save your .prproj files, since selecting “Save As” does not default to the current project location as it should.  Instead it defaults to the folder containing the last piece of media that you imported.  I usually follow a strict project organizational pattern, but I have saved projects in the wrong folders daily since switching to CS5 thanks to this “feature.”  Hopefully this can be fixed in a minor update.

Now a subsidiary application to Premiere Pro, Encore CS5 is the next logical step in that application’s evolution as an interactive authoring tool.  The most significant new feature for larger productions would have to be support for DDP output, so that your Encore projects can now be replicated at a professional facility without any special hardware required on the authoring side.  Adobe has also continued to develop the Flash export options, allowing Encore users to output their interactive experiences directly to the web, now with most of the BluRay interactivity features, including the new multipage menus, supported as well.  Encore projects are now completely cross platform compatible, between the Mac and PC versions.  There is also better support for 23.976p and 24p source files, and an easier subtitling process.  AVCHD files can now be authored directly to BluRay discs without any transcoding, decreasing processing time, and increasing output quality.

Previous versions of Encore have used Dynamic Link to import Premiere Pro sequences as source clips, and more significantly in my opinion, to import After Effects compositions as motion menu backgrounds.  I was never really that impressed by Dynamic Link for Encore source clips, because your Premiere Pro sequence is usually finished by the time you make a DVD, and since you are going to have to transcode to MPEG2 at some point anyway, little time is saved.  Motion menu design on the other hand seems like a much better fit for the workflow benefits offered by Dynamic Link, since the menu needs to be edited in both Encore and AE during the authoring process.  Encore now also uses Dynamic Link to offload transcoding of those sequences, or any other incompatible source footage, to Adobe Media Encoder.  This will free up Encore to continue interactive authoring work, while asset transcoding proceeds in the background, and also allows third party accelerated encoding plug-ins that are compatible with Adobe Media Encoder, to be utilized for these internal transcodes.  While I am unaware of any CS5 compatible encoders, this was an issue with the Elemental Accelerator in CS4.  The files it generated at 24p were re-transcoded in Encore anyway, and Encore couldn’t utilize the accelerated plug-in directly.  Anyhow, while no CS5 version of Elemental Accelerator has been announced, Matrox’s CompressHD should be able to accelerate Encore’s BluRay H.264 encodes, once Matrox releases CS5 compatible drivers.

The one thing still missing from the Adobe package is a good intermediate format for exchanging files between other applications, systems, or facilities.  This capability is available from 3rd parties like Cineform or Matrox, but until Adobe integrates support for one directly into the suite, there will not be a single universal standard format, that can be counted on to be compatible everywhere.  It needs to be compatible with both Mac and PC, and store at least 10 bit HD footage with sufficient compression to playback smoothly on a laptop.  Apple and Avid have both recognized this need, and developed ProRes and DNxHD respectively.  AVC-Intra is Adobe’s current recommendation to meet those workflow needs, but that format has a complicated file structure, and is not a codec optimized for smooth playback.  This universal format would tie in well with OnLocation, if Adobe ever added support for HD-SDI capture capabilities.  That would allow ingest, either live or from tape playback, into a compressed format for editorial, with log notes entered in real-time similar to their current tapeless solution options.  And if a professional selection of scopes could be viewed of the SDI input signal, you would have an all-in-one onset workstation solution.  OnLocation CS5 currently offers most of these capabilities, but is limited to HDV, XDCam, and P2 acquisition for most of its features.  Oh, and if it supported my Canon DSLR, that would be pretty cool too; maybe next time.

As a whole the CS5 release is a major turning point for Adobe, especially with the problems their users experienced with CS4.  While I don’t yet fully utilize many of the new features, CS5 has already revolutionized the way I work.  Premiere Pro CS5 has replaced After Effects CS4 as central application from which I manage my projects.  I was lucky enough to be on the beta team, so most of the major issues I encountered with the software in my specific workflow, have already been resolved during the development process.  While CS5 is by no means perfect, it is a huge step forward that will begin to rebuild user confidence in Adobe products.  Once third party hardware and plug-ins have had time to be updated to be compatible with the new 64bit native applications, I see no other disadvantages or caveats to upgrading to CS5, which I highly recommend.

FTC Disclosure: I have been on Adobe’s beta team for many years, and Adobe has provided me a copy of CS5 for this review.  My only admitted personal bias is my preference of Windows over OSX, because I like full control over every aspect of my computing experience.  If for some reason that bothers anyone, there are plenty of other sources of information on the internet, but I try to provide unique insight on how each of these tools fits into the larger post-production picture.  Any relevant critique or response is welcomed.

So once the lenses issue was dealt with, were left with a selection of AVCHD encoded MOV files.  AVCHD is a processing intensive format that does not playback or edit very well.  While Quicktime would play the files, it clipped the blacks and the whites at incorrect levels.  16 and 235 were being stretched to 0 and 255 on decode, lowering the dynamic range.  This was caused by Quicktime incorrectly interpreting one of the header fields on the file.  The solution to this was to use CoreAVC to decode the files when converting into a different, and ideally more edit friendly, compression format.  Shortly after this workaround was developed, Apple released a Quicktime update (7.6) that fixed this particular issue entirely.

Beyond the clipping issue, there are other tricks to maximize the dynamic range of the 5D.  The picture style is used to control the way that the camera converts the 14bit RAW still into an 8-bit JPEG.  The same picture profile settings are applied to the 8-bit recorded video.  This allows you to do things to get the maximum detail out of the available 8-bits of color depth.  The first few projects I worked on that used the 5D, we used a custom picture profile that I got from Stu Maschwitz’s ProLost blog, High Gamma 5.  We did a number of comparison tests, and while High Gamma 5 gave us a wider total dynamic range, for our feature film, we eventually decided to use Neutral, one of the default Canon presets.  Neutral gave us a file that was closer to the final look we were going for, and with only 8bits of color depth, burning in your look, at least to a degree, should result in better picture quality at the end of the day.

Every file the camera records is named MVI_####.mov, with an auto-incrementing number, and no real override options.  That makes things simple on tiny projects, with one camera since each file has a unique name.  On larger projects, and ones that use more that one camera, (We usually have 15) file management can be a bit more work, to keep things straight throughout the post production process.  Our solution was to rename each MOV file with a unique 8 digit identifier as the new filename, and store the key to the original card and filename in a database.  This allows each clip to have a consistent name throughout the process, to show up on EDLs as a tape name or clip name as desired, without truncating unique values after the 8th digit for certain formats.  By the time we are done ,we usually have a source MOV, an Avid MXF, and an online Cineform AVI, all with the same content and file name.

Next up was the framerate problem, at 30p.  The first few projects I did with 5D we posted at 29.97, so the issue was solved with a simple reinterpretation of the framerate, when converting from the source AVCHD into an editing codec, and tweaking the audio .1% to match.  Unfortunately 29.97 footage doesn’t intercut with film very well, and won’t print back for theatrical masters either, so sometimes a 24p workflow is required.  For 24p projects, the conversion solution is much more complicated, involving motion compensated frame blending.  After extensive testing we concluded that this was best done with the Revision Effects Twixtor plugin for AE, or using Optical Flow in FCS Compressor on OSX.  Having a PC centered workflow, I favor the AE based solution.  With render times at around an hour per minute of source footage, it is impractical to convert all of the source footage on large projects, which necessitates an offline edit.  Since we don’t have timecode and keycode, relinking for the online requires a bit more creativity.  We have found some interesting options that are unique to Premiere Pro CS4, related to the way it links EDLs to existing source footage that make this much simpler than our first tedious tests, which involved manually rebuilding projects at 24p back in Premiere Pro CS3.  The new CS4 version can convert the TC-In on an EDL to a framecounted In-point of an existing media file, with makes the onlining of 5D footage a relatively simple automatic process after a few find-replace edits (.mov to .avi in our case) to the EDL.  In the future, it looks like Canon is going to support 24p recording on all of their DSLR offerings, so all of these crazy 30p workarounds will soon be an obselete thing of the past.

Although it is much better in rough environments than most other electronics, Canon DSLRs do have their weaknesses.  I have operated a 5D in temperatures of 20 below zero, and in the desert at over 120 degrees fahrenheit.  While we had no issues in the cold, where solid state recording has a huge advantage over tape, there are some issues at higher temperatures.  The camera sensor itself is a large piece of silicon, that generates a lot of heat on its own, and when combined with a high external temperature, in the worst cases is shuts off the camera.  You probably have to be over 150 degrees to get to that point, leaving the camera in a black metal box in direct sunlight for an extended period of time, but we have seen it happen.  A much more frequent problem, that is harder to detect, is that as the sensor begins to overheat, there will be much more video noise in the recorded picture, especially in the darks.  This is probably due to a higher latent voltage on the chip as its electrical resistance changes with the temperature increase.  This has only been a problem for us when shooting with the same camera for many hours in a hot environment, and our solution is usually just to swap the camera bodies for one that has not been used in a while.  This obviously requires having multiple cameras on set, which isn’t always an option on lower budget projects.

The last issue, that we are still finding new ways to deal with, is rolling shutter.  Having a large format CMOS sensor, DSLRs are subject to rolling shutter, or inconsistencies between the top and bottom of the frame.  I have spent the last few months working on a project that put the 5D into some of the most intense situations.  As a fairly lightweight device, it is subject to more jitter and shake than a larger camera with more inertia, and with the camera moving, the rolling shutter results in the recorded picture being slightly geometrically skewed, depending on the direction of the motion.  We also shoot high speed objects, like helicopter rotor blades, which are known to cause strange artifacts in certain instances.  So far we have been lucky with that, and haven’t found any of those types of issues in our footage.

The type of rolling shutter artifact we are struggling with the most, is gunfire muzzle flashes, especially at night.  In the dark, the flash blows out the imager, but the flash does not last as long as even a single frame.  So with the rolling shutter, the top half of a frame will be totally blown out, with the bottom part looking normal, because the flash had subsided by the time that part of the chip was sampled, or vice versa.  Setting the shutter speed lower than the frame rate causes it to screw up more of the frame or frames, and setting it higher causes it to narrow the flash into a distinct horizontal band in the footage, neither of which is desirable.  One thing we have found that helps is setting the shutter on the 5D to 1/30th.  (We usually set it to 1/50 to get similar motion plur to film shot with a 180 degree shutter)  With the 30p framerate, the flash either affects an entire frame, or matching parts on two subsequent frames.  (Bottom part of one frame, and the reverse area on the top of the next one)  This gives us an entire over exposed frame if we stitch the two parts together.  This can be hand cut back into footage that has been brought from 30p to 24p by manually selecting frames.  It remains to be seen if this solution can be scaled practically to our entire movie.  The best way to avoid this issue is to avoid recording gunfire at close range in very dark environments.  The farther you are from the muzzle flash, and the more ambient light there is, the less it is going to flare out your camera, minimizing the degree of the resulting rolling shutter artifact.

So that should convey some of the challenges are in faced in using DSLRs for filmmaking, especially on large scale projects, but it is by no means an exhaustive list.  As the tools evolve to suit the cameras, and the cameras evolve to suit the tools, many of these issues will become much easier to solve and require fewer workarounds.  The AVCHD decoding issue was solved by a new release of Quicktime, the manual lense control was solved with a new firmware release from Canon.  The 30p conversion process is the next issue I see becoming a thing of the past, if Canon can get a 24p recording option onto the 5D.  I am looking forward to that day, but in the mean time I have 2TB of 30p footage, divided into 5,000 shots, to cut into a 24p film, so I have a lot of work ahead of me.

NVidia and Elemental Technologies report that the accelerator can provide up to a 10x increase in encoding speeds, but mileage may vary in real world use.  With my system, I usually found encoding rates to be 50% to 100% faster with real world work, depending on the output settings, which is still a significant improvement if you do a lot of exporting.  The relative increase in encoding performance that you will experience depends on the speed of your system.  Older systems will see a major improvement, while newer high end workstations are already quite fast, so the change will be less dramatic.  My 3Ghz 8-Core workstation can already encode my HD timelines to MPEG2 for DVD faster than realtime, but with the Elemental Accelerator, I was able to cut the time for a two minute export from 1:27 to 30 seconds.  On a two hour clip, that would be a thirty minutes instead of an hour and a half.  My 1080p H.264 encodes for BluRay output only saw a 50% increase in encoding speed.

Exports of more complicated timelines see less of an improvement because render speed is not accelerated by the GPU as much as encoding speed is.  If you coming from a single flat clip, the increase in export speed will be much more apparent.  I don’t use AVCHD footage, but when encoding from those types of source files, the GPU can accelerate the processing intensive AVCHD decode, as well as the MPEG2 or H.264 encode.  This should lead to more dramatic performance improvements, especially on laptops, where available CPUs are not as powerful as their desktop counterparts.  When I first reviewed the accelerator software earlier in the year, I anticipated that laptops would be even better served by a GPU based accelerator than desktops.  If you frequently make these types of exports from a laptop with a Quadro GPU, this plugin will be worth it.  In the desktop world it isn’t as simple, since the CPUs are not so weak by comparison to the GPU. If you make DVDs or BluRays for a living it will be a no brainer, but otherwise it all depends on how much of your day you spend waiting for exports to complete.

There are still a few other issues to be worked out.  The Adobe Media Encoder CS4 is designed to minimize the decrease in system performance during background rendering by pausing exports during timeline playback.  This may be necessary on lower end systems, but eight-core workstations should have power to spare, before we even factor in the GPU.  This prevents Adobe Premiere CS4 from really utilizing the CPU power being freed up by the GPU acceleration, because any time it is called upon, the render gets paused anyway.  Adding the option to disable that functionality would be beneficial to both Elemental Accelerator users, and anyone else with a high-end workstation, so that they could truly multi-task their system.  Until then, its probably best to work outside of Premiere Pro during your accelerated exports, to truly take advantage of your available system power.

With NVIDIA’s Digital Video Pipeline in the works, its easy to see where this could be going.  Once they have SDI video inputs available, we could see realtime capture directly to H.264 and MPEG2 files, among many other things.  With professional I/O, the available GPU power could be used to turn the video card into a full editing accelerator card.  With SLI Mosaic and other new developments with the QuadroPlex, I can see it being scaled up to 4K frame sizes with realtime performance, and I am definitely looking forward to that.

Other new features in Photoshop CS4 that require OpenGL acceleration will benefit work on even regular sized images.  Images viewed at magnification levels that are not even multiples (1/2, 1/4, etc.) are now displayed at much higher quality, as well as being much faster and more fluid, since the preview is being scaled in the graphics card.  You can smoothly zoom to any level instead of the previous default 100%, 50%, 33%, etc. and the image will look perfectly clear.  You can also rotate the previewing plane without permanently effecting the image quality or dimensions, and continue to interact with the image at that angle.  Although it initially seems trivial, after a bit of thought I can conjure up a variety of important but obscure uses for this capability, most of which relate more to art and design than video post-production.  There is also a new pixel grid that is available to clearly dilineating the boundaries between individual pixels at extremely close zoom levels (>600%)  I am honestly not sure why this requires GPU acceleration, but it is unavailable unless you have a supported OpenGL graphics card.  My only complaint about the new functionality and acceleration found in Photoshop CS4 is that Adobe saw fit to specifically skip its implementation in the 64bit version of Windows XP.  Since this is the highest performing version of Windows available, Adobe’s failure to directly support it will cause its users continued frusteration for some time to come.

The only possible relief on the horizon in that regard is that Windows 7 seems to have been fast-tracked by Microsoft due to the Vista issues.  Speaking of future developments, hopefully we will soon be able to display 10bit color from Photoshop onto a Dreamcolor LCD via DisplayPort.  I suspect that this may not be available until CS5, but I am sure it is coming, based on the increasing level of hardware support on products that are now coming to market, with the Quadro CX leading the way.

The CS4 applications that will see significant performance gains from hardware acceleration, are After Effects, Photoshop, and Premiere Pro.  The improvements in After Effects and Photoshop will also be evident with any other previous generation high end GPU, while the new hardware accelerated H264 encoding support for Premiere Pro is specifically tied to the new Quadro CX card.  NVidia has also recently announced the Quadro FX 4800, with basically identical hardware specifications to the Quadro CX, and which retails for about $200 cheaper.  The extra cost is buying you access to the CUDA based RapiHD H.264 encoder, that is available in no other form besides in conjunction with the Quadro CX card.  If you have no need for accelerated H.264 encoding, you could consider saving $200 with the FX 4800, but I envision the possibility of NVIDIA releasing more CX-only tools for creative professionals, since that card is targeted towards that specific market.  Not to be overlooked, NVIDIA has also released the Quadro FX 5800, with an incredible 4GB of memory, but that should only be needed by applications with the most intense processing requirements, and is a class above NVIDIA’s previous Quadro products.  The Quadro CX is compatible with the same HD-SDI output daughter card that the previous Quadro FX 4600 and 5600 cards used, for broadcast and post-production applications.  Hopefully we will see more software applications directly supporting that interface card in the near future.  The Quadro CX is based on the same core architecture as the new GTX 200 series of consumer cards, while the Quadro FX 4600 was based on the same technology as the GeForce 8800GTX, which is now two generations out of date.  It is to be expected, that there is an all around performance increase with the new cards in almost any application, but Adobe has been specifically adapting their software to leverage the power of these graphics processors.

Of the many applications in Adobe’s new CS4 Suite, After Effects is the one that most fully and effectively integrates the power of GPU acceleration to increase processing power and application responsiveness.  Many of these features are not new, but by nature of the way they are designed, grow more powerful as GPU performance increases.  OpenGL allows most of the 3D processing required for advance compositing to be offloaded to the GPU for dramatic increases in performance and responsiveness.  There are also many plug-ins and effects that specifically take advantage of GPU power.  Synthetic image generation like noise and fractals, as well as artificial 3D blurring are some of the best fits for effective GPU acceleration in AE.  Many of these improvements are only implemented for faster previews unless the user specifically selects OpenGL exporting, due to a possible loss in quality based on the lack of precision of OpenGL, but certain effects such as the new “Cartoon” vectorizing filter, that experiences dramatic (30x) rendering improvements with GPU processing, utilize GPU acceleration both for rapid previews and for accelerating the final export render.  In most of the synthetic object (Noise, fractals, shapes, blurs, etc.) previewing tests I did in AE, OpenGL acceleration with the Quadro CX provided a ten fold increase in performance over CPU based rendering.  This is the difference between an interactive experience, and a plan-next-move-while-rendering workflow.  While any graphics card with OpenGL support can accelerate processing in After Effects, as compositions and projects become larger and more complex, the benefits of the Quadro CX’s increased memory and processing power will come into play.  Complex projects will experience a greater increase in performance than simpler ones, when upgrading to a higher end GPU.

The next program in the CS4 suite to experience major performance improvements through GPU acceleration is Photoshop CS4.  Most of these improvements come from newly added implementation of OpenGL processing, and therefore, like After Effects, they are not specifically tied to the new Quadro CX.  But the power of the new Quadro CX makes the benefits of these improvements more dramatically obvious, especially on larger images.  These improvements in Photoshop are fairly extensive, and I will review them in detail in the next posting in this series.  As far as the Quadro CX is concerned, with its large 1.5GB cache of onboard memory, it is more than capable of handling the largest and most complex operations that almost anyone would attempt in Photoshop.

Adobe’s Premiere Pro CS4 also takes advantage of the GPU in a few less significant ways, for basic effects.  The one totally new aspect that the Quadro CX brings to the table is accelerated encoding, specifically encoding to H.264 with the new RapiHD encoder from Elemental Technologies.  This is the primary marketing piece specific to the CX card, and my third and final post in this series on the Quadro CX will be about CUDA and its implementation in this new encoder.

After all of the Adobe tests, I had two other programs that utilize the GPU that I wanted to try on this new high end card.  The first was Iridas SpeedgradeDI, which is specifically programmed to run on NVidia’s Quadro cards.  The base version running with DVI or 8bit DisplayPort attached monitors worked great, and nothing I could do with my limited knowledge of the program could even get it to drop a frame.  The real test for that application would involve connecting the optional SDI daughter card for true 10bit output.  Hopefully the 10bit color depth supported by the DisplayPort will eventually allow that level of monitoring without the high priced SDI daughter board.  The highest-end customers will still require an SDI output in order to use SDI interfaced external waveform and vectorscope tools, or live broadcast outputs.

Lastly, I ran my favorite program, Battlefield 2 to test out the card’s 3D rendering capabilities.  BF2 was released over three years ago, and therefore is not a cutting edge test, but it is my most recent high performance game.  With all of the settings maxed out, at maximum resolution on my 30″ LCD, I was able to get 99.9 FPS about 90% of the time, with the occasional dip into the low nineties for complex scenes.  Anyhow, the Quadro CX should be more than up to the task for those late night “stress relief” sessions with any modern 3D “application” if desired.

Once I had thoroughly tested the Quadro CX’s acceleration capabilities, I endeavored to verify the capabilities of it’s newly supported output interface.  I hooked my HP Dreamcolor LCD to the card via the new DisplayPort interface, hoping to get some taste of 10bit color output.  Unfortunately, currently none of the major applications I currently have installed are programmed to take advantage of this capability.  I do have a small utility from NVidia that displays 16bit TIFF files in 10bit color depth, and I can confirm that yes, there is a difference, and yes, the combination of the Quadro CX and the HP Dreamcolor does give you full hardware support for 10bit color display.  Hopefully in the future we will see updates and plug-ins that will unlock this feature in useful ways.  There is a 10bit capable SDI plug-in that NVidia released for After Effects 7 quite a while back, as a simple demonstration of their new SDI capability, and I am hoping to see an equivalent DisplayPort version for both After Effects and Photoshop, especially since NVidia and Adobe seem to be working together more closely these days.

Anyhow, if you are in the market for a new high end GPU, the Quadro CX has all of the processing power that most people could possibly need.  At its currently available price of about $1800, it has directly replaced the Quadro FX 4600, and no question is a superior product.  Now if you already have a 4600, the jump to the CX is not immediately necessary unless you are currently pushing your system to the limits, or you encode a lot of footage to H264.  The Photoshop and After Effects GPU support in CS4 will work nearly as well with a 4600, but there is a difference.  With any other previous generation card, you should see a significant all around performance increase with any application that leverages GPU processing power.

That information should give you a good general idea of what the new Quadro CX card is capable of, and I will be detailing the improvements in Photoshop CS4 and the RapiHD encoder in my upcoming posts.

Originally the only way to generate Cineform AVI files at this resolution was to scan film to DPXs, and then convert those image sequences, presumably in After Effects.  Interestingly, while being at 2K resolution, these files were being downsampled to YUV colorspace, which improved compression rates and performance, at the expense of the quality certain color information.  Eventually the SI-2K camera provided a second root source of Cineform2K imagery, that did not involve converting from an uncompressed source file at any point in the process.  I anticipate we will see 2K data being captured over SDI directly to Cineform files in the near future, if they are not doing so already, as both AJA and BMD have products with the required hardware capability.

The SI-2K brought with it one other new option to the format, which was an alternate pre-debayer RAW ‘colorspace’ instead of the YUV data that was normally being compressed.  CineformRAW video files had even better compression ratios, but the only way to take advantage of the benefits of pre-debayered images was by tapping directly into a single sensor imager, as the SI-2K did.  This eventually led to a third underlying option, with the development of Cineform 444.

Cineform444 allows full RGB encoding, processing, and mastering of image data, which is the norm in a digital intermediate environment.  The RAW files of the SI-2K had the required information for RGB processing, but it hadn’t been fully taken advantage of until that point.  DPX filmscans have full RGB info, as do most still images and CGI files.  Only ‘broadcast video’ files and formats are limited to YUV.  While increasing file sizes and processing requirements, this RGB processing brought the “Cineform Intermediate” to the next level quality wise.  Cineform now shows off their format in comparison the the popular HDCam-SR format, which doesn’t directly compare since SR can’t be accessed natively, but it communicates the idea to people who still think in terms of tape formats.  I believe SR tapes that are recorded in 444 RGB can be captured directly to Cineform444 via BMD hardware, but I have not had the opportunity to test that myself.  I have worked on projects where we captured 444 RGB data uncompressed through a BMD Multibridge, and then later converted parts to Cineform444 to work with.  I have also extracted CineformRAW files from the SI-2K directly to full range Cineform444 files.  Either way, this workflow has allowed me to work on visual effects shots at full 2K resolution in RGB, on my system at home without a dedicated RAID, and even on my laptop if I could suffer the 12inch screen size.

Around the same time that the Cineform444 capabilites were being developed, the option to include an imbedded alpha channel was added, and the maximum bit depth was increased from 10bit to 12bit.  I have yet to take advantage of the 12bit capability, but the embedded alpha channel can be useful for everything from animated titles to visual effects.  I find the biggest gains from this development when using Cineform for my After Effects work.

Cineform has also been working to develop a cross-platform solution, so that Cineform compressed media can be easily shared between Macs and PCs.  Their first step was to develop a Quicktime wrapper for their original AVI codec.  Next, they ported the compression codec to OSX, assisted I am sure, by Apple’s switch to Intel processors.  The last steps remaining to be completed are an accelerated CineformRT engine for the new Mac version of PremierePro CS3, and an OSX port of their HDLink conversion tool.  At this point Cineform is the only reliable compressed format for transfering video between Windows and OSX systems, besides saving to a DPX sequence, which has obvious disadvantages.

As mentioned in an earlier post, Cineform has recently released a beta of their next improvement to the capabilities of their format, with support for 4K imagery, or higher.  Although 4K filmscans could be used as source, their immediate target is data from the Red One camera, with 4K output from the RedCine conversion utility.  With wavelet compression for efficient reduced resolution decoding, RGB colorspace, 10 or 12bits of color depth, support for alpha channels, and cross platform compatibilty, Cineform seems well positioned to meet the needs of this emerging market.

When used with just the sensor, the SI-2K is an amazingly small and light imaging device for use in the tightest places, but it must be tethered to a computer system within 100 yards.  Usually the lens is much larger than the imager.  If an all-in-one form factor is desired, there is a full-sized camera body that can be attached to it, which contains a fully functional PC, running Windows XP Embedded.  The data is then saved on removable hard disks, that can be hot swapped via the internal USB interconnect.  With 300GB 2.5″ hard disks available now, you should be able to get 4 or 5 hours of footage onto a single drive.

Once the Cineform RAW footage is on disk, it can be edited in that codec on most popular NLEs with Cineform’s Neo2K product.  For best performance, you can use Prospect2K in Premiere Pro to edit with accelerated playback and rendering.  The RAW files are debayered in real time by the software, and edits or exports are rendered to regular Cineform 422 or 444 files, which work seemlessly together.

The SI-2K, when processed at the highest quality, generates a 10bit log RGB image at up to 2048×1152 pixels.  The colorist I work with has been very pleased with the dynamic range the images retain by the time they reach him in color correction.

Its small size gives this camera a few unique advantages.  Like the Iconix, it can capture stereoscopic footage without the use of a beam splitter, since the cameras can be mounted so closely together, side by side.  ParadiseFX has also recently developed an interesting application for the cameras, in their 18K wide 360 degree surround 2K360 video capture system, utilizing nine cameras running in parallel.  It has very limited and specialized uses, but will be much cheaper than any similar solution.  The next logical step is the inverse of this, with a ring of cameras pointing inwards, allowing time to be frozen while the perspective changes, like in the famous shots from The Matrix.

There are a few issues to be dealt with in using the camera though.  No matter how you configure the capture setup, you are using a Windows PC as your capture device.  This entails almost all of the same possible pitfalls of any other direct to disk recording system.  Your camera has to “boot up” and could lockup, crash, or even theoretically get a virus, if you network it to offload capture files.  Eventually you will probably have to reformat and reinstall the software as performance declines.  Silicon Imaging says they are currently working on a USB Flash based solution to that problem.  On the positive side, it makes it much easier to update the drivers and software, to add functionallity to the system as it continues to be developed by Silicon Imaging.  For example support to capture directly to Quicktime wrapped .mov files was recently added.

This is all around, a very creative and innovative imaging solution from Silicon Imaging, but they are still developing it into a smoother running product.  It has many possible uses that it is uniquely well suited for, but one needs to be ready to compensate for its limitations as well, as with any imaging solution.

The HDLink is a small box that converts HD-SDI video signal to DVI or HDMI, for monitoring video signals at full resolution with a computer LCD screen or projector.  It supports regular 1080 and 720 in most variations, and RGB 444 over dual link SDI.  It also has stereo sound output for audio monitoring.

These HDLinks are in widespread use at a number of the facilities I work at, and at under $500, they are within the reach of independent editors or anyone else who needs an economical solution for monitoring SDI.  They do get hot though, so they should be powered down when not in use, and they don’t last forever.

The new HDLink Pro adds 3Gb/s SDI for 2K support on a single SDI cable, but lacks dual link support, which is still the most popular RGB standard.  It also adds 6 Channel Audio de-embedding to analog RCA ports for 5.1 surround sound monitoring.  I have not yet had the opportunity to try one of these out yet, but I look forward to doing so as I find myself doing more 2K work, especially now that I have a 30″ LCD to connect it to.

 Separately but related, Blackmagic’s Workgroup Video Hub is a 12×24 SDI router.  It supports SD and HD, but currently not the 3Gb/s SDI standard that Blackmagic has been rolling out this year.  With a bit of creativity, it can easily support dual link connections, but that lowers the number of devices you can connect by half.  All patching is controlled in software via USB, which is connected to a single system, that hosts a network utility to share control with all users.  Their utility for patching as well designed and fleshed out at this point.  You can also patch through a built-in downconverter to SD, and it has a Still Store, ideal for color bars or logos.

We use these at three facilities I work at, and they bring a dramatic increase to the flexability and efficiency of your workflow.  They are not economical for use by a single individual, but will be well worth it in work areas shared by 3-10 people.  They greatly ease the sharing of high end video equipment.  For example, I can output any edit workstation’s SDI playout to either: a projector on an HDLink, an LCD on an HDLink, an HD CRT, our vectorscope, or any combination thereof.  Using a Video Hub can have a significant effect on your environment if you frequently find yourself reconfiguring your peripherals when switching between different projects.

The one Axio feature that I use the most, is Matrox’s somewhat proprietary HD compression format, MPEG I-Frame HD.  It supports full size and frame rate HD files at a constant bitrate, between 50 and 300Mb/s, depending on your needs.  I use 100 or 150Mb/s for projects going directly to broadcast video, and 50Mb/s is useful for offline edits that will be reconformed uncompressed.

Matrox also allows native editing of many different formats that Premiere does not otherwise support, including DVCProHD, P2 MXFs, XDCam MXFs, and HDV including 24fps.  In theory these all play in real time on the same timeline.  Real world results are a bit different, and I have experienced many performance problems when mixing these formats, but individually they all play back great.  My usually solution to this mirrors the Cineform workflow, to convert everything into one high performance codec, specifically Matrox’s MPEG I-Frame HD codec.  Matrox’s wide file support allows me to make all of these conversions without leaving Premiere, although using a separate project for asset preparation is highly recommended.  This conversion process is also accelerated by Matrox’s faster than realtime exports of all supported formats.

Matrox has done extensive work to allow their video effects to be accelerated by the system’s video card GPU, on Axio LE systems.  There are a limited number of supported GPUs, but they are budget friendly consumer gaming cards, as opposed to expensive workstation OpenGL certified cards.  This allows motion, opacity, advanced color correction, and many other effects to be processed in real time.  This realtime performance is their primary selling point, and allows for a lot of creative freedom.  Combined with native imports and realtime exports, this makes Axio a very efficient editing system as long as you aren’t trying to work too far outside the box.

The MPEG I-Frame HD codec is hardware accelerated in all applications, but has very poor performance in systems without an Axio installed.  Matrox sell the M.key, a USB dongle to unlock that codec on systems without Axio Hardware, but using it for anything besides rendering to or from MPEG I-Frame HD will result in poor performance.  On the other hand, MPEG I-Frame HD is the only COMPRESSED codec I have ever gotten to playback in realtime with Iridas’ SpeedGradeDI software, but only with the Axio card installed of course.  Matrox uncompressed files can be read on any PC using their free VFW codecs.  The fact that all supported compressed formats are limited to 8bit color is a significant limiting factor, as well as memory issues in larger projects. Timelines with more than about 10 minutes of edited material begin to exhibit performance problems, and until native 64bit processing allows for addressing greater than 4GB of Ram, that will continue to be an issue.

Axio is a highly accelerated editing platform for HD 422 work, but is not well suited for long form projects.  It is not going to help you online your film at 2k or 4k, or even process RGB data at 444 at all.  It does however support 24fps footage and editing, so it could work well as an offline editor. Offlining your short film on Axio at 24fps offers some interesting advantages when onlining in Premiere Pro with a different solution, like Cineform 2K or Blackmagic RGB.  All titles, motion effects, and other Premiere specific edits should pass through automatically to the conform.  Axio will also probably be the fastest way to edit your 30 second spot, or 10 minute corporate training video, after shooting on HDCam or HDV.