ImageType used by ImageData

Question

I have an image of type "Byte" (from ImageType[image]). When I perform calculations with ImageData[image] (like Total[ImageData[image]]) are these calculations still performed with the "Byte" data format or conversion into another format like Integer or Real is performed under the hood?

Answer 1

ImageTypes "Bit", "Byte" and "Bit16" correspond to images encoded by 1-bit, 8-bit and 16-bit unsigned integers, images of types "Real32" and "Real" are encoded by single- (32-bit) and double-precision (64-bit) real numbers correspondingly.

According to the Documentation page for ImageType (under the "Backgound" section):

Many image processing functions automatically convert integer-type images to a real type for higher accuracy prior to computation.

So in general image-processing function may convert images into "Real32" or "Real" type for processing. Under the "Details and Options" section on the Documentation page for ImageData we read:

For images of type "Byte" or "Bit16", ImageData[image] always normalizes values to lie between 0 and 1.

This statement is ambiguous because it is not clear whether the channel values are rescaled to run from 0 to 1 or they are simply converted into a real type via dividing by 255 (for "Byte") or 65535 (for "Bit16"). It is also not clear which real type is the target: "Real" or "Real32".

Let us investigate. For this purpose we generate an image of type "Byte" where channel values run from 100 to 200 (the full allowed range is from 0 to 255):

SeedRandom[1]
data = RandomInteger[{100, 200}, {125, 325, 3}];
i = Image[data, "Byte"]
ImageType[i]

"Byte"

Now we check what ImageData returns:

Head@ImageData[i][[1, 1, 1]]
MinMax[ImageData[i]]
% * 255

Real

{0.392157, 0.784314}

{100., 200.}

We see that ImageData by default converts the channel values of images of type "Byte" into Real numbers between 0 and 1 without rescaling. To prevent the conversion we need to specify the Automatic type of image:

ImageData[image, Automatic] is equivalent to ImageData[image, ImageType[image]].

Checking:

Head@ImageData[i, Automatic][[1, 1]]
MinMax[ImageData[i, Automatic]]

Integer

{100, 200}

Let us ensure that the original image data are converted into "Real" type, not "Real32":

Max[Abs[ImageData[i, Automatic]/255. - ImageData[i]]]

1.11022*10^-16

This difference corresponds to MachinePrecision computations:

10^-$MachinePrecision

1.11022*10^-16

So the channel values are converted from "Byte" into Real numbers with MachinePrecision. Let us try another conversion method:

Max[Abs[Divide[ImageData[i, Automatic], 255.] - ImageData[i]]]

0.`

With this last method the difference is explicitly zero, so ImageData seems to use the intelligent Divide-based approach for the data conversion.

---

Another thing which is worth to know is that ImageData by default has Interleaving -> True:

ImageData // Options

{DataReversed -> False, Interleaving -> True}

This means that ImageData by default returns interleaved data even for images without interleaving:

data = RandomReal[1, {3, 4, 5}];
image = Image[data, Interleaving -> False];
ImageData[image, Interleaving -> False] == data == Transpose[ImageData[image], {2, 3, 1}]
Transpose[data, {3, 1, 2}] == ImageData[image]

True

True

---

And a little aside from the main topic but still worth to cite here a comment by Szabolcs:

Unfortunately the documentation does not spell this out, but Real and "Real32" (i.e. floating point) types do not clip in Image. Integer pixel types do clip. It is unfortunate that this is not made clear in the documentation because it is one of the first things that is explained on WRI image processing seminars ...

˂The Documentation for Image˃ says that values are displayed from 0 to 1, but it does not say that greater or smaller values are not stored. It says that values are coerced to the target type. This coercion necessitates clipping with integer types, but not (usually) with floating point type (unless you go above ~10^300). The same with rounding. Basically it says that M does whatever is needed to fit the value into the target type.

Here are the corresponding statements under the "Details and Options" section of the Documentation page for Image:

• Image[data] by default allows any real number, but displays only values between 0 and 1.

• Image[data,"type"] can be used to create an image of a specified data type. Values in data are coerced to the specified type by rounding or clipping. By default, "Real" is assumed.

Checking:

ImageData[Image[{{-10, 0, 1, 100}}], Automatic]
ImageData[Image[{{-10, 0, 255, 1000}}, "Byte"], Automatic]
ImageData[Image[{{-10, 0, 65535, 100000}}, "Bit16"], Automatic]

{{-10., 0., 1., 100.}}

{{0, 0, 255, 255}}

{{0, 0, 65535, 65535}}

Conversion to Raster and backwards:

Show[Image[{{-10, 0, 1, 100}}]][[1, 1]] // Normal

{{-10., 0., 1., 100.}}

ImageData@Image@Show[Image[{{-10, 0, 1, 100}}]][[1]]

{{-10., 0., 1., 100.}}

Answer 2

As you can see in its documentation, you can specify what format ImageData should use to return its output. Let's use your avatar as an example:

img = Import["https://i.stack.imgur.com/6m6HE.jpg?s=328&g=1"];

ImageType[img]                       (*Out: Byte        *)

Total[ImageData[img, "Byte"], -1]    (*Out: 42543966    *)
Total[ImageData[img, "Bit16"], -1]   (*Out: 10933799262 *)
Total[ImageData[img, "Real32"], -1]  (*Out: 166839.     *)

As a further example, you can inspect the different ways a pixel's color values are reported:

ImageData[img, "Byte"][[150, 150]]    (*Out: {75, 77, 74}                   *)
ImageData[img, "Bit16"][[150, 150]]   (*Out: {19275, 19789, 19018}          *)
ImageData[img, "Real32"][[150, 150]]  (*Out: {0.294118, 0.301961, 0.290196} *)