Hello, I'm Nguyen Trung Kien, an AI engineer at Ridge-i. Today, I would like to share information about color spaces for color augmentation techniques. Additionally, I will demonstrate the color augmentation techniques based on the basic color space as well as some advanced ones.
Introduction
Color augmentation is a useful technique for increasing the diversity of the training data, thus, helping the model generalize better to different lighting conditions and color variations. In general cases, a simple color technique can work, however, in some special cases, when the image includes structure (edges and contours) and the foreground color looks similar to the background, it may not be suitable as it may not preserve the apparent detailed structure of the image by changing the color. In such scenarios, using color augmentation in an advanced color space can provide a more suitable solution. In this post, I will use satellite images [1] as an example for demonstrating the color augmentation effect.
Color space
Before introducing color augmentation techniques, I would like to introduce briefly some definitions of color spaces for the following color augmentation methods.
What is color space
A color space is a specific organization of colors that helps in the representation, reproduction, and manipulation of colors in digital imaging. Different color spaces cover different ranges of colors. Below I will introduce some of the existing color spaces that have been widely used.
RGB color space
The RGB color space uses the primary colors red, green, and blue, which correspond to the three types of color-sensitive cone cells in the human eye. These cells respond to long (red), medium (green), and short (blue) wavelengths of light, allowing any color to be represented by mixing varying intensities of these three colors. There are several RGB color spaces, below are 2 common ones:
Linear RGB: Linear RGB values are raw data obtained from a physical sensor. R, G, and B values are directly proportional to the amount of light that illuminates the sensor.
Standard RGB (sRGB): sRGB values apply a nonlinear function to linear RGB values. Most of the images we use are in sRGB format because the nonlinear function was optimized for the display (such as CRT, and LCD display), therefore, they appear brighter and the colors are easier to distinguish. The figures below show how to transform between linear RGB and sRGB [2] (below C represents R/G/B value scaled into [0, 1]).
HSV color space
HSV color space is a cylindrical representation of RGB color space. Color in HSV space is represented by a combination of hue (H), saturation (S), and brightness value (V). In color theory, hue is a fundamental property of color, often represented as an angle where 0 degrees corresponds to red, 120 degrees to green and 240 degrees to blue. This angular representation allows for the visualization of colors in a circular spectrum, covering the full range of visible hues. Under the same lightness and saturation, we can change the hue value to provide different colors. Therefore, we can manipulate colors more easily, which is better for tasks like color correction and artistic design. The transformation between sRGB and HSV can be found in [3].
The transformation can also be done easily with OpenCV
import cv2 # convert image from sRGB to HSV img_hsv = cv.cvtColor(img_rgb, cv.COLOR_RGB2HSV) # convert image from HSV to sRGB img_rgb = cv.cvtColor(img_hsv, cv.COLOR_HSV2RGB)
CIE XYZ color space
Although sRGB is used widely, it has a limitation that it does not cover all visible human colors. On the other hand, XYZ, a color space published by the International Commission on Illumination (CIE) can cover almost the human vision color [4]. Unlike RGB, which is based on the primary colors red, green, and blue, the XYZ color space is derived from three hypothetical primaries: X, Y, and Z. In this system, Y corresponds to brightness or luminance, Z is primarily sensitive to shorter wavelengths, and X responds to both shorter and longer wavelengths of light. The figure below shows some color gamut areas including sRGB as a triangle, and XYZ as a horseshoe shape.
Image source: https://www.benq.com/en-us/knowledge-center/knowledge/color-gamut-monitor.html
We can easily convert from Linear RGB to XYZ with simple matrix multiplication, and convert it back by multiplying to the corresponding inverse matrix.
Note that, the above formula is used for Standard Illuminant D65, in which white color is represented as [X, Y, Z] = [0.9505, 1.0000, 1.0890]. Also, there are several transformation matrices based on precision level, and the above matrix was the one published in [2]. Below is the Python code using OpenCV for transformation.
import cv2 # convert image from Linear RGB to XYZ img_xyz = cv.cvtColor(img_rgb, cv.COLOR_RGB2XYZ) # convert image from XYZ to Linear RGB img_rgb = cv.cvtColor(img_xyz, cv.COLOR_XYZ2RGB)
CIE Lab and CIE LCH
CIE Lab
The CIE Lab color space is a color space defined by the CIE in 1976, in which L defines lightness (black at 0 and white at 100), a represents green/red, b represents blue/yellow, with negative values toward green or blue and positive values toward red or yellow correspondingly. Both a and b are unbounded, but will typically be clamped to the range [−128, 127] as 8-bit integers. One of the advantages of the CIE Lab color space over RGB is its design for perceptual uniformity [5, 6]. This means that a numerical change in CIE Lab coordinates produces a consistent perceptual color difference, whereas in RGB, the same adjustment does not result in a uniform perceptual shift. CIE Lab color space cannot be transformed directly from sRGB, hence, we need to convert it into XYZ first. The transformation between XYZ and Lab is described here [7].
OpenCV provides a function for completely transforming between Linear RGB and Lab.
import cv2 # scale down the image for non-linear transformation img_rgb *= 1./255 # convert image from Linear RGB to CIE Lab img_lab = cv.cvtColor(img_rgb, cv.COLOR_RGB2Lab) # convert image from CIE Lab to Linear RGB img_rgb = cv.cvtColor(img_lab, cv.COLOR_Lab2RGB) # scale up the image img_rgb *=255
CIE LCH
CIE LCH is the cylindrical model of CIE Lab equivalent to HSV of RGB in which LCH represents Lightness, Chroma, and Hue (this Hue is similar to HSV Hue, but red is 24 degrees, green is 162 degrees, and blue is 246 degrees [6]). To convert from Lab to LCH, we retain the L value from Lab and transform Cartesian coordinates (a, b) to polar coordinates. In the Lab color space, both a and b components represent color, requiring adjustments to both to manipulate the overall color. Converting to the LCH color space simplifies this process, as we can modify the hue (H) directly to achieve a new color, eliminating the need to adjust the (a, b) pair separately.
Limitations
Even though the CIE Lab color space provides a better representation of human vision, it still has limitations in its ability to accurately predict hue. It lacks luminance-level dependency, making it unable to account for luminance-related effects. Additionally, CIE Lab does not incorporate background or surround dependency, making it not able to predict simultaneous contrast or account for changes in image contrast. Furthermore, CIE Lab lacks mechanisms to model cognitive effects, such as compensating for the illuminant, which is crucial in cross-media color reproduction. Lastly, CIE Lab does not provide correlations for absolute appearance attributes like brightness and colorfulness [6].
OKLab and OKLCH
OKLab is a newer color space than the CIE Lab developed to address the unstable hue problem and maintain visual consistency with human color vision [8]. The meaning of Lab in OKLab is the same as in CIE Lab, except that they use a different method to convert from XYZ to OKLab. The transformation is slightly more complex but can be implemented using matrix multiplication, as described in [8]. OKLCH is also defined as the cylindrical model of OKLab, which allows for easier hue adjustments. OpenCV does not support OKLab but the Colour Library can be used instead, which also supports XYZ, CIE Lab, and HSV transformation.
import colour # convert image from sRGB to XYZ img_xyz = colour.sRGB_to_XYZ(img) # convert image from XYZ to OKlab img_oklab = colour.XYZ_to_Oklab(img_xyz) # convert image to LCH img_oklch = colour.models.Jab_to_JCh(img_oklab) # convert LCH to OKLab img_oklab = colour.models.JCh_to_Jab(img_oklch) # convert image from OKLab to XYZ img_xyz = colour.Oklab_to_XYZ(img_oklab) # convert image from XYZ to sRGB img_rgb = colour.XYZ_to_sRGB(img_xyz)
Color augmentation techniques
In this section, we will examine the color augmentation results based on the above color spaces, along with their limitations.
Channel permutation
The definition of channel permutation color augmentation involves shuffling the RGB channels of an image. As the values of the R/G/B channels change, a new image with potentially different colors could be generated. Although channel permutation can only provide five additional data variations and the colors cannot be controlled, it is still used as a data augmentation method in some experiments [9]. Below is an example of channel permutation.
Hue adjustment with HSV color space
This method is the most common color augmentation technique and is widely used in many applications. This technique involves converting the RGB image to HSV color space, adjusting the H(hue) value by a specified amount, and then converting the image back into RGB to generate a new color version. This technique can be applied to any image to achieve different color variations. However, in some specific cases, it may distort the image's appearance.
In the above satellite image [1], the green and gray colors represent the land, while the blue represents the sea. The different blue colors are related to the sea depth level. Visually, the difference in color among sea areas when the hue is adjusted by 0.375 becomes less differentiated. There are 2 possible reasons for this situation:
The sensitivity of the human eye depends on the type of color. With the same difference value between two RGB colors, it can happen that the blue region appears distinguishable, whereas the green region does not.
By definition, when applying hue adjustment in HSV color space, the maximum and minimum values among RGB are kept the same, and the remaining value is adjusted in the range of [min, max]. And then based on the new hue, RGB values are permuted to form a new color variation. If the input image has a structure of low foreground-background contrast, certain hue values may further reduce this contrast.
Left: original image, middle: hue adjustment with hue=0.5, right: hue adjustment with hue=0.8375
In these examples, we divided the image into three vertically stacked sections. The top and bottom parts (backgrounds) have the same color RGB values while the middle part has slightly different values. We perform a hue change in HSV color space from the original image (the most left image) into different colors (middle and right images). In the above examples, the most left image shows that the sum of the absolute differences in RGB values between the middle section and the background is 7. And if we apply hue adjustment with hue=0.5, the difference becomes 5, which is less distinguishable.
Hue adjustment with CIE Lab color space
To avoid the above issue, we can adjust the hue in the CIE Lab color space. The steps are the same as hue adjustment with HSV, but we use either CIE Lab or OKLab as the target color space instead.
In the above examples, we can see the sea area in images applying hue adjustment in the CIE Lab and OKLab color space is more distinguishable than the ones in HSV with the same amount of hue adjusted.
Examples with three vertically stacked sections also show the advantage of OKLab over CIE Lab and HSV.
In the examples, hue change with HSV has a low RGB difference, while both hue changes with CIE Lab and OKLab have higher differences. However, in the example of CIE Lab, all the red channel values become 0. This occurs because the CIE Lab color space has a wider gamut. The corresponding color in CIE Lab is outside the sRGB gamut, so the values are clipped to 0. The hue change with OKLab is the best among the three cases as it can either preserve the human visual or maintain the color similar to the original one.
Below is the Python code for adjusting the hue in the OKLab color space.
import colour
import numpy as np
def adjust_hue(img, hue=0.0):
"""Adjust hue with OKLab color space.
Args:
img: sRGB image is scaled in the range [0, 1]
hue: The amount of hue change, can be either negative or positive. The default is 0.0
Returns:
New sRGB image
"""
img_xyz = colour.models.sRGB_to_XYZ(img)
img_oklab = colour.models.XYZ_to_Oklab(img_xyz)
img_oklch = colour.models.Jab_to_JCh(img_oklab)
img_oklch_aug = img_oklch.copy()
img_oklch_aug[:,:,2] = (img_oklch_aug[:,:,2] + int(hue*360))%360
img_oklab_aug = colour.models.JCh_to_Jab(img_oklch_aug)
img_xyz_aug = colour.models.Oklab_to_XYZ(img_oklab_aug)
img_aug = colour.models.XYZ_to_sRGB(img_xyz_aug)
img_aug = np.clip(img_aug, 0, 1)
return img_aug
Summary
In the previous sections, I provided an overview of color spaces and their corresponding color augmentation techniques. Each technique comes with its own set of advantages and disadvantages, which were illustrated using both synthetic and real-world image examples. Although the above demonstration used satellite images may not be the best use case as we do not often change the color of satellite images, it can reflect the difference.
Depending on your data, you can choose the most appropriate technique. If your data is simple, color augmentation in the HSV color space can be sufficient. However, if your data is complex and your task involves human vision, advanced color enhancement using OKLab may be a better choice. Besides the introduced color spaces, there are many other choices supported by the Colour library such as HunterLab, ICtCp, IPT 2021, ...
The advantages and disadvantages of the introduced color augmentation techniques are described in the table below.
| Technique | Advantages | Disadvantages |
|---|---|---|
| Channel permutation | - Preserve the RGB ratio as in the original - Fast computation |
- Cannot provide many colors |
| Hue adjustment (HSV) | - Provide many colors - Simple calculations for conversion |
- Appearance of color shade structure changes - Not aligned with human visual perception |
| Hue adjustment (CIELab) | - Provide many colors - Preserve color shade structure - Better aligned with human visual perception |
- Require complex computation |
| Hue adjustment (OKLab) | - Provide many colors - Preserve color shade structure - Enhance alignment with human visual perception |
- Require complex computation - Not supported by OpenCV |
References
- Open Access Hub
- sRGB
- Zou, Yuxiang & Zou, Xiangqun. (2023). Fast Algorithm of Brightness and Contrast Enhancement Based on HSV. 10.3233/FAIA230915.
- https://www.rp-photonics.com/color_spaces.html
- LCH colors in CSS: what, why, and how? • Lea Verou
- Mark D. Fairchild. (2013). Color Appearance Models. 10.1002/9781118653128
- CIE 15:2004 3rd Edition
- A perceptual color space for image processing
- Yoo, Jaejun & Ahn, Namhyuk & Sohn, Kyung-Ah. (2020). Rethinking Data Augmentation for Image Super-resolution: A Comprehensive Analysis and a New Strategy. 8372-8381. 10.1109/CVPR42600.2020.00840.
