Introduction

Do cameras see the same color as us? Can cameras always accurately reproduce colors that our eyes see? This interactive tutorial explores these questions and many more interesting aspects of camera raw color space. In particular, we will walk you through an important concept in both color science and camera signal processing: color correction, the process of correcting the color perception of a camera such that it is as close to ours as allowed. In the end, you will get to appreciate why you should never trust the color produced by your camera and how you might build your own camera that, in theory, out-performs existing cameras in color reproduction.

Caveats. 1) This tutorial demonstrates the principle of color correction with many important, but subtle, engineering details omitted; we will mention them when appropriate. 2) Color correction is one of the two components in camera color reproduction, the other being white balance (or rather, camera's emulation of chromatic adaption of human visual system). We have a post that discusses the principles of chromatic adaptation and its application in white balance. The relationship of color correction and white balance is quite tricky, but Andrew Rowlands has a fascinating article that demystefies it for you.

Step 1: Exploring Camera Color Space

In principle, cameras work just like our eyes. Our retina has three types of cone cells (L, M, and S), each with a unique spectral sensitivity, translating light into three numbers (the L, M, S cone responses) that give us color perception. Similar to the three cone types, (most) cameras use three color filters (commonly referred to as the R, G, and B filters), each with a unique spectral sensitivity and, thus, also translate light into three numbers. In this sense, you can really think of a camera as a "weird" kind of human being with unconventional LMS cone fundamentals. Not all cameras use three filters though. Telescope imaging cameras use five filters, just like butterflies!

The left chart below shows the measured camera sensitivity functions of 48 cameras in two recent studies. The 48 cameras are classified into four categories: DSLR, point and shoot, industrial cameras, and smartphone cameras. The sensitivities are normalized such that the most sensitive filter (usually the green filter) peaks at 1. What should be noted is that the sensitivities functions measured here are not just the spectral transmittances of the color filters; rather, they are measured by treating the camera as a black box, and thus reflect the combined effects of everything in the camera that has a spectral response to light, such as the anti-aliasing filter, IR filter, micro-lenses, the photosites, etc.

The default view shows the average sensitivities across the 48 cameras, but you can also select a particular camera from the drop-down list. As a comparison, we also plot the LMS cone fundamentals in the same chart as dashed lines. As is customary, the LMS cone fundamentals are, each, normalized to peak at 1. As is usually the case in color science, these normalizations merely introduce some scaling factors that will be canceled out later if we care about just the chromaticity of a color (i.e., the relative ratio of the primaries).

You can see that the shapes of the camera sensitivities functions more or less resemble those of the cone fundamentals, which is perhaps not all that surprising. After all, cameras are built to reproduce colors that our eyes see. The shapes, however, are not an exact match. Most notably, you will see that the L and M cone responses overlap much more than the R and the G filters overlap. As you can imagine, sensitivities that are overly close to each other won't provide a great ability to distinguish colors. In the extreme case where two filters' sensitivities exactly match, the camera is dichromatic.

Although by default disabled, the chart also contains the CIE 1931 XYZ Color Matching Functions (CMFs). You can enable them by clicking on the legend to the left of the chart. In fact, clicking on a legend label toggles the visibility of the corresponding curve in the chart. The XYZ CMFs are just one linear transformation away from the LMS cone fundamentals, and is used as the "common language" in colorimetry when comparing different color spaces. So we will use XYZ as the connection color space in the rest of the tutorial.

In the same way we can plot the spectral locus in the XYZ color space, we can also plot the spectral locus on a camera's native RGB color space. The 3D plot on the right shows you the spectral locus on the XYZ, LMS, and the camera's RGB color space. The LMS locus is by default disabled, but you can enable it by clicking its legend label. Drag your mouse and spin around the 3D plot. You can see that the spectral locus in XYZ and in camera's RGB space do not match.

You can also draw your own camera sensitivity functions. Select "Custom" from the drop-down list, and start dragging your mouse. As you draw, the spectral locus in the camera RGB space will be dynamically updated on the 3D plot on the right. You can hit the Draw Reset button to clean the drawing.

Step 2: Problem Formulation

The fact that the locus in XYZ and in the RGB locus do not match means that each spectral light has different XYZ and RGB tristimulus values. That in itself is not a problem: the LMS and the XYZ tristimulus values of spectral lights don't match either. Critically, however, the LMS and XYZ color spaces are just one linear transformation away from each other. Any light, not just spectral light, can be converted between the XYZ and the LMS space using one single $3\times 3$ matrix multiplication. So it really doesn't matter in which space you express the color of a light; whether in LMS or XYZ (or any other colorimetric space), it's all the same underlying color.

Here comes the central question of this tutorial: is a camera's RGB color space also just a linear transformation from the XYZ/LMS space? If so, then the raw camera RGB values of any light can be translated to the correct XYZ values of that light, essentially recovering the color of the light from camera captures. In other words, the camera sees the same colors as us, and metamers to our eyes are also metamers to the camera. In general, if the camera raw color space is precisely a linear transformation away from the XYZ color space, the camera is said to satisfy the Luther Condition and that the camera color space is colorimetric (in that we can use the camera to measure color). What would happen if a camera doesn't satisty the Luther Condition? Well, lights that are different to us (i.e., have different XYZ values) might be the same to the camera (i.e., have the same raw RGB values), and vice versa. Metamers to the camera would not be metameras to our eyes.

Stated mathematically, we want to find a $3\times 3$ transformation matrix $T$ that satifies the following equation:

$ \begin{bmatrix} X_0 & X_1 & X_2 & \dots \\ Y_0 & Y_1 & Y_2 & \dots \\ Z_0 & Z_1 & Z_2 & \dots \end{bmatrix} = \begin{bmatrix} T_{00} & T_{01} & T_{02} \\ T_{10} & T_{11} & T_{12} \\ T_{20} & T_{21} & T_{22} \end{bmatrix} \times \begin{bmatrix} R_0 & R_1 & R_2 & \dots \\ G_0 & G_1 & G_2 & \dots \\ B_0 & B_1 & B_2 & \dots \end{bmatrix}, $

where $[X_i, Y_i, Z_i]^T$ is the XYZ values of a light and $[R_i, G_i, B_i]^T$ is the camera raw RGB values of the same light. Since the transformation matrix $T$ has 9 unknowns but $T$ should ideally work for any arbitrary light, i.e., infinitely many equations, the system of equations is over-determined. Instead of finding an exact solution, we will instead settle for a best-fit $T$ that works well for a set of lights whose color reproduction we care about.

Step 4: Pick Illuminant

Step 5: Calculate Color Correction Matrix

With the illuminant selected, we now have everything to Generate Equations to concretize the linear system:

$\begin{bmatrix} \boxed{??} \\ \boxed{??} \\ \boxed{??} \end{bmatrix}$ $\small{ = \begin{bmatrix} T_{00} & T_{01} & T_{02} \\ T_{10} & T_{11} & T_{12} \\ T_{20} & T_{21} & T_{22} \end{bmatrix} \times}$ $\begin{bmatrix} \boxed{??} \\ \boxed{??} \\ \boxed{??} \end{bmatrix}$

The first matrix has 24 columns, each representing the XYZ values of a ColocChecker patch under your chosen illuminant ${\boxed{??}}$. The last matrix also has 24 columns, each representing the RGB values of a patch in the camera RGB space under the same illuminant. We get an over-determined system with 72 equations and 9 unknowns.

Before solving the system, let's look at the patches in both the XYZ and the camera RGB spaces, which is shown in the left chart below. Evidently, the XYZ and the RGB values of the patches do not overlap, but if you spin around the chart you'll likely see that the relative distributions of the patches within the XYZ and the RGB space are roughly similar, suggesting that a linear transformation might exist.

Step 6: Interpreting Camera Color Space

With the transformation matrix $T$ calculated, we can use it to get more insights into a particular camera's color space. The TL;DR is that a typical camera's color space contains imaginary colors. While the observations we are about to make are generally true, but let's take one specific camera just so we can be concrete. Click the Choose iPhone 12 Pro Max button to choose iPhone 12 Pro Max as the target camera and D65 as the illuminant. Everything on this page will be updated accordingly.

Differences Between Skin Tones

If we use the human skin tones as the correction target, we'll see that darker skins generally have worse color correction results compared to lighter skins, as it evident in the heatmap above, where bottom rows represent darker skin tones and have higher $\Delta E^*$ errors. Seemingly neutral correction algorithm, as is implemented in this tutorial, results in racial bias manifested as different color reproduction accuracy measures across skin tones. There is a great talk by Theodore Kim on how graphics and camera imaging research could, independent of any individual intent, contain racial biases.

Note that this is not to say that the color correction algorithm used in a particular camera will always produce the exact bias as bad as discussed above. Our intention is to provide a concrete example where camera imaging could inadvertently introduce racial bias.

Are Spectral Lights Corrected?

The first question you can ask is, does $T$ correct colors of other lights that are not used for color correction? In particular, we are interested in how the spectral lights are corrected. Go back to the spectral locus plot in Step 1 and click the "Draw Corrected Locus" button, which will plot the transformed spectral locus (the "XYZ (Corrected)" curve). Ideally, you want that locus to precisely match the actual XYZ spectral locus, but most likely you will see that that are way off. This is perhaps not all that surprising, because the spectral lights are not used for calculating the best-fit matrix.

We can gain a better understanding of the correction results in the chromaticity plot. The chart below plots the "ground truth" spectral locus and the 24 patches in the xy-chromaticity plot, and then overlays the reconstructed locus and patches transformed from the camera RGB space using matrix $T$. Hover on any point will show the coupled counterpart as well. Notice how the original 24 patches and the reconstructed ones are almost overlapped (because they are used to estimate $T$), but the true and corrected spectral locus are miles apart. Those reconstructed spectral colors, when converted to an output-referred color space, will give you incorrect colors.