Image Information Metrics

Information Capacity and more

Introduction – Summary of the information metrics – Video – Documentation – Updates – White papers –
Documentation web pages – Conferences – EI 2024 paper – AutoSens Detroit 2024 talk –
ISO 23654 – Call for participation

Introduction

The market for cameras that produce images for Machine vision (MV) and Artificial Intelligence (AI), in contrast to pictorial images for human vision, is steadily growing. Applications include automotive (driver assistance and autonomous vehicles), robotics, security, and medical imaging systems.

Questions arise when designing camera systems for such applications.

How best to select (or qualify) cameras for MV/AI applications?
Are there fast and convenient measurements that are good predictors of machine vision/AI system performance (better than traditional measurements such as sharpness or noise)?
What image processing (ISP or filtering) is optimal?

To answer these questions, we must go beyond standard measurements of sharpness (MTF or SFR) and noise and apply metrics derived from information theory, including information capacity and related metrics for object and edge detection.

These metrics are important because Object Recognition (OR), MV, and AI algorithms operate on information, not pixels, making them far better predictors of system performance than MTF or noise.

Imatest has developed highly convenient methods for measuring information metrics. The white papers (with varying degrees of detail) describe how the new metrics can be used to select (or qualify) cameras and determine the optimum Image Signal Processing (ISP) for Object Recognition, which is likely to improve the performance of MV and AI algorithms. The new metrics are included in the Imatest 24.2 release.

A quick summary of image information metrics

Classic measurements of sharpness (MTF or SFR) and noise are not sufficient for predicting machine vision (MV) system performance.

Image information metrics, based on Shannon’s information theory, which are far more predictive of MV system performance, can be measured from standard slanted-edge test images with contrast ratios of 10:1 or less. The ISO 12233-standard 4:1 contrast ratio is strongly recommended. The most important image information metrics are

Information capacity — an indicator of a camera’s potential performance. It is unchanged by reversible, linear image processing.
Ideal Observer Signal-to-Noise Ratio (SNRi) — a measure of the detectability of an object. It increases with object size, and is affected by image processing.
Edge location standard deviation (σ) — a measure of how well the position of the object can be located. It is relatively independent of object size and is also affected by image processing.

A matched filter defines the optimum image processing for a specific measurement and object. Practical filter (equalizer) design involves tradeoffs (not severe) for the range of objects that need to be detected.

Additional metrics used to derive the above three are Spatially-dependent noise power (N(f)), Noise Power Spectrum (NPS(f)), and Noise Equivalent Quanta (NEQ(f)). Information metrics share a common kernel, \(K(f) = MTF^2(f) / NPS(f)\). Information metrics are the basis of the ISO 23654 standard, now under development.

The best technical introduction to the new metrics is on the revised Electronic Imaging 2024 paper, “Image information metrics from slanted edges.“

Norman Koren will present a two hour short course,
Information Metrics for Optimizing Machine Vision Systems

at the Electronic imaging conference in Burlingame, California (near SFO), on February 5, 2025, 3:30-5:30PM.
He will be available during the conference to answer questions and discuss the new metrics in depth.

Video: Image Information Metrics in Imatest

Documentation, white papers, and instructions

Updates — We continue to learn new things about the information metrics.

The units of Edge SNRi, which are frequency = 1/distance (typically 1/pixel), are non-intuitive and difficult to understand. To remedy this we have defined a new metric, Edge Location Standard Deviation (σ) = 1/Edge SNRi, which has units of pixels. It is the standard deviation of any point of the estimated edge location.

The key metrics used to quantify machine vision system performance are Mean Average Precision (mAP) and Intersection over Union (IoU). We will be working on verifying the hypothesis that SNRi is a predictor of mAP and Edge Location σ is a predictor of IoU.

We will be making some changes in the description of the noise calculation. We now look upon spatially dependent noise N(x) as the first step in the noise calculation, and the calculation of the Noise Power (Wiener) Spectrum, NPS(f), as the second step. The NPS(f) calculation is required to fully characterize the noise, which must be normalized so that ∫N(x)dx = ∫NPS(f)df .

We are updating the documentation, but as of late June, 2024, it’s a long, tedious process. Many of the changes have been incorporated into the Autosens Detroit 2024 keynote talk (PDF).

White papers

The white papers linked below contain similar material.
They range from concise and technically lightweight for managers and non-technical readers
to detailed and comprehensive (but challenging to read).

Introduction to Image Information Metrics is a concise introduction to image information capacity and related metrics, with a minimum number of equations and technical detail. We may shrink it further for use in a brochure for marketing.

Image Information Metrics and Applications: Reference (41 pages) has all the equations and technical detail, but it may be challenging to read. It’s the primary reference for the metrics.

Image Information Metrics in Imatest is an intermediate-level introduction to image information capacity and related metrics with moderate technical detail. We are deprecating it to reduce the amount of redundant documentation.

Camera Information Capacity from Siemens Stars (2020) describes a method that is slower and less convenient than the slanted edge, but better for observing the effects of image processing artifacts such as demosaicing, data compression, etc.

Imatest software documentation web pages

Image information metrics from Slanted edges: Instructions – instructions on the new calculations.
Image information metrics from Slanted edges: Equations and Algorithms – A reference for the instructions page. Equations for image information metrics, conveniently measured from any slanted-edge, including NPS, NEQ, and SNRi. Similar to the white papers, which will be kept more up-to-date.
Shannon information capacity from Siemens stars This calculation from 2020 needs to be reviewed and updated. Some corrections may be needed.
Meaning – Results – Summary

Conferences (newest first)

Electronic Imaging 2025, February 5, 2025, Burlingame, California (near SFO).

Norman Koren will present a two hour short course, Information Metrics for Optimizing Machine Vision Systems.

Autosens Europe 2024 — Barcelona

Norman Koren, gave a three-hour tutorial on image information metrics. [Autosens website]

Autosens USA 2024 — Detroit

The keynote talk, New camera quality measurements for optimizing Machine Vision systems (slides in PDF format), updated to include Edge location standard deviation (σ), is available for downoad here.

Electronic Imaging 2024

Norman Koren presented a one-hour keynote talk on Image Information Metrics (covering much of the material in the white papers) at Electronic Imaging 2024 in Burlingame, California.

The revised and updated EI2024 paper, “Image information metrics from slanted edges,” is an excellent introduction to the new metrics. It contains an improved explanation of the noise measurements and a description of Edge Location standard deviation (σ). Available for download here. Recommended

The published version (from Imaging.org; not up-to-date) is available for download here.

The slides from the talk (PDF format) are available for download here.

International Standard ISO 23654 (in development)

ISO 23654 began development in early 2023. Based on ISO 12233, it will specify how to calculate information capacity and related metrics from slanted edges. It is in the PWD (preliminary working draft) phase. It is open to new members.

Call for Participation

We are interested in working with machine vision and computer vision experts who are experienced with studying how image quality metrics correlate with object detection performance in computer vision systems. Please contact infocap@imatest.com if you are interested.

A small experiment in color appearance, based on the above tables. The colors of the central rectangles are… (you guessed it).