One of the important aspects of the INNOISP is its software-based modular structure. The ISP performance and the influence of adding or removing each of ISP block can be simulated and visualized through the software version. The ISP design will be fixed once desired image quality and system performance is achieved. For instance, Figure 1 illustrated a rendition of a scene with a full-stack ISP.

​​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Figure 1: Rendered image using a full-stack ISP.​

​

The same RAW file can be processed through another ISP version that is optimized for a linear sensor by removing the expensive LTG (Lens Transport Generator) / GTM (Gamma Tone Mapping) ISP block:

1- Signal decoding and unpacking

2- AWB/AE
3- Black level correction
4- Lens Shading Correction

5- Demosaic
6- Color Correction
7- Gamma and YUV conversion

8- YUV Denoise/sharpening

9- Output formatter

 

The proposed ISP block diagram and the rendered image are shown in Figure 2.​

​

​

​

​​

​

​

​

​

​

​

​

​

​

​

​

​​​​​

 

 

 

 

 

 

 

 

 

 

Figure 2: Top- ISP block diagram for a linear sensor, fixed focus wide FoV lens. Bottom- rendered image.

​​

Ultimately, The ISP blocks can be further reduced for a compact low power or high-frame rate design suitable for AI-based applications.

1- Signal decoding and unpacking

2- AWB/AF
3- Black level correction
4- Demosaic

5- Gamma and YUV conversion

6- Output formatter

The selected ISP layout and sample processed image are illustrated in Figure 3.

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Figure 3: Top- ISP block diagram for a compact design with hardware limitations. Bottom – rendered image.

​

ISP Blocks

Below is a brief explanation of the implemented ISP blocks. Each ISP block functionality and algorithm will be explained in the full-version section of this document.

 

Signal Decoding and Unpacking

This step is to arrange the bits from the camera to form a correct pixel representation. Depending on the communication interface the data might be compressed, packed, or with a predefined pedestal. This block also handles signal combination in case of multi-frame captures for multi-exposure HDR image acquisitions. The output of this block is the linearized signal that represents the scene radiometric quantities.

 

Bad Pixel Correction

This block is to fix the defected pixels on the sensor using the neighboring pixel data. The defected pixel detection and correction is either based on a static pixel map or an adaptive algorithm where the defected pixel is detected and fixed dynamically.

 

Lens Shading Correction (LSC)

LSC corrects for the lens color shading and vignetting by applying proper weights to di[erent color channels. The scene statistics for AWB and AE algorithms are extracted after the image corrected for LSC.

 

Auto White Balance (AWB)

AWB block simulates human vision color adaptation by weighing the RGB signals for a given light source color temperature. The white point is extracted using a weight-map in R/G-B/G color space and is modulated by scene brightness. The AWB block can be bypassed when the white point weight are preferred to be fixed.

​

Auto Exposure (AE)

Auto exposure is one of the key ISP blocks that adjusts the sensor ISO and integration time for an optimum exposure. An optimum exposure can be very challenging in WDR scenarios that is a common case for automotive industry.

 

Auto Focus (AF)

To modulate the lens focal distance based on the estimated object distance using contrast or PDAF-based methods.

 

LTM/GTM

This block is to applies a global and/or local non-linear transformation to each pixel based on adjacent pixels, which is particularly essential for the HDR sensors. It maintains the local contrast and enables the system to perform in lower bit-depths without compromising image contents. This step can be removed for LDR sensors saving processing and complexity.

 

Demosaic

Demosaic or de-Bayering refers to the algorithms to extract RGB signals from an under- sampled Bayer image. It requires three- or five-line bufers depends on the chosen demosaicing algorithm.

 

Denoise/sharpening in Bayer and YUV domains

This block consists of a variety of gaussian and convolution kernels and bilateral filters that enhance noise and sharpness using weighted average of the local pixel information. Both denoise and sharpening blocks can be applied in Bayer domain or in the YUV image format. The Bayer domain usually provide more objective resutls since the signal is in its raw format. YUV denoise/sharpening applies more aggressive denoising and sharpening on the image chrominance luminance channels according to the human visual system and perception.

​

Color Correction (CC)

Color correction refers to a transformation that converts camera-dependent color coordinates to the standard sRGB color space. It usually consists of a 3x3 matrix multiplication for RGB input signal. The more comprehensive LUT version can be implemented when more accurate color reproduction is needed or the sensor spectral sensitivity differs from human visual system.

 

Gamma

Gamma is a non-linear transformation prior to luminance-chrominance YUV color space. It is an essential block to prepare the output signal format and cannot be removed.

 

YUV Color Space Conversion

This block transforms the RGB image into an opponent color space YUV or YCrCb for more e[ective perceptual sharpening/denoising and signal compression. This block also follows video streaming standards (e.g. Rec 601 or Rec 709) to save some computations in the next steps.

 

Color Fringing Correction

Color Fringing refers to the color bleeding artifact that is visible on the high contrast edge. This artifact can be further reduced by combining an adaptive edge detection and color desaturation in the chrominance color planes.

 

Output Formatter

This is the last block on the ISP pipeline that encodes the signal according to the output single capture or video streaming standards. The out put can be in standard YUV444, YUV422 or YUV420 format for further compression end video encoding.

​​​

Contact us for detailed information - sales@innowave.design