Bayer demosaicing in Nuke

Get the Nuke gizmo or check out the BlinkScript code

Achieving photorealism is the ultimate goal of most CG/ VFX projects. This becomes obvious when looking at the progression of lighting and shading models in the recent past. However, apart from simulating physically plausible light interactions there are many subtle effects that are no less vital to making an image more photo-realistic. We are used to see minor imperfections in the lens or the sensor like distortion, aberration, flares and noise. They can add a lot as long as they stay hidden in plain sight.
Exploring the simulation of these phenomena has always fascinated me for that reason. One such effect that is not commonly used is demosaicing. Understandably so since in a CG context throwing away two thirds of the information you rendered seems counterintuitive. Still, to me it seemed worthwile to experiment with since almost any digital camera nowadays shows this kind of artifact.

A digital camera sensor consists of an array of cells which measure the intensity of incoming light. To capture color values you need to take three measurements per array position – for R, G and B separately. Since its not possible for three sensors to occupy the same space simultaneously there is a need to overcome this limitation. Approaches like 3 CCD solve it by splitting light into its components with a prism and then recording it with three separate sensors. While providing good quality its main drawback is the amount of space needed to house the sensors. Considering the importance of sensor size especially in digital cinematography this is a problem.
Another way to solve this is alternating the wavelength each cell records in a pattern along the surface of the sensor, creating a Color Filter Array. The arrangement of color filters usually follows the pattern of a Bayer Matrix, where the sensor is split into 50% green, 25% red and 25% blue cells. As the artifacts this array is prone to are easily overcome with increasing resolution and available space is less of a problem, this method is most widely used today.

Bayer BGGR CFA

So there are two components to this. First we need to define the CFA and then reconstruct the missing information from the CFA image (demosaicing). The bayer pattern is easily rebuilt in nuke with the modulo function:

2x upscaled detail shot of the bayer pattern

There are many approaches to demosaicing, starting with simple bilinear interpolation and ending at expensive pattern recognition methods. Since this is more of an experiment I settled with a basic three-pass method that uses a weighted median to retain image sharpness. A good explanation of the method I implemented can be found here.