7
$\begingroup$

I need to run an exported NetChain in ONNX, surprisingly the model exports and runs successfully, but there's a discrepancy in the image loading stage (the NetEncoder'ing that must be written manually).

Here's a minimal working example of the issue:

Export["oxford.jpg", ExampleData@ExampleData[{"TestImageSet", "Oxford2011"}][[1]]];
encoder = CloudGet @"https://www.wolframcloud.com/obj/8af91972-cf2f-4754-9ea7-34d53d8cb312";

correct = encoder @ Import @ "oxford.jpg";
byHandMMA = ImageData[ImageResize[Import@"oxford.jpg", {224, 224}], 
                  Interleaving -> False] - {.48, .46, .4};

Max @ Abs[correct - byHandMMA] (* ok, not zero but small enough... *)

enter image description here

Then here it is in Python:

# pip install scikit-image 
import numpy as np
from skimage.transform import resize
from skimage import io

img = io.imread("oxford.jpg")
img = np.rollaxis(img, 2, 0) # uninterleaved
img = resize(img / 255, (3, 224, 224), anti_aliasing=True)
img[0]=img[0]-0.48  
img[1]=img[1]-0.46  # subtracting mean channel values (from NetEncoder)
img[2]=img[2]-0.4
img = img.astype(np.float32)
img

Importing back yields a big difference with what it should be:

byHandPy = Normal[%];
Max @ Abs[correct - byHandPy] (* 0.125127 *)

enter image description here

Here's another attempt with a different image library:

# pip install Pillow
import numpy as np
from PIL import Image

img = Image.open('oxford.jpg').resize((224, 224))
img = np.array(img, dtype='float32')/255
img = np.rollaxis(img, 2, 0) # uninterleaved
img[0]=img[0]-.48  
img[1]=img[1]-.46  # subtracting mean channel values (from NetEncoder)
img[2]=img[2]-.4
img

Now the error is smaller, but still too big and impacts the network's performance.

byHandPy = Normal[%];
Max @ Abs[correct - byHandPy] (* 0.0627451 still too big*)

enter image description here

Perhaps someone knows exactly how NetEncoder["Image"] works (exact resizing method, order, antialiasing options etc..)? I think I'm missing some tiny detail.

Refs:

$\endgroup$
4
  • 1
    $\begingroup$ @xslittlegrass Could you help here? $\endgroup$
    – user5601
    Commented Nov 30, 2021 at 6:28
  • 1
    $\begingroup$ NetEncoder["Image"] uses OpenCV to load images. The details are hidden in OpenCVLink binary though. The actual call to link is in OpenCVLinkImportOpsCVLoadImagesFromPath. The following code reveals the call: ` Trace[encoder[file], OpenCVLinkImportOpsCVLoadImagesFromPath[___]] ` $\endgroup$ Commented Dec 11, 2021 at 8:09
  • $\begingroup$ @PavrlPerikov Can you turn this into an answer? $\endgroup$
    – user5601
    Commented Jan 25, 2022 at 1:24
  • $\begingroup$ I provided the answer with some warning below $\endgroup$ Commented Jan 27, 2022 at 7:16

1 Answer 1

5
$\begingroup$

According to @user5601 suggestion I'm turning my comment into an answer.

NetEncoder["Image"] uses OpenCV to load images. The exact implementation is hidden inside OpenCVLink binary library though and deeper in libopencv_core.dylib. The actual OpenCV call is cv::imread in case you use it as in NetEncoder["Image"][File["...."].

The following snippet reveals the code that calls OpenCVLink:

encoder=NetEncoder["Image"];
file = File[...];
Trace[encoder[file], OpenCVLink`ImportOps`CVLoadImagesFromPath[___]]

WARNING:

Please be aware of the bug in NetEncoder["Image"][File[...]] implementation which presents since early 12.x releases (and will be fixed in 13.1): it leaks every image it loads so using it repeatedly in affected versions is impossible — you'll run out of memory pretty soon.

$\endgroup$
2
  • 1
    $\begingroup$ That's an insanely bad regression $\endgroup$
    – M.R.
    Commented Feb 1, 2022 at 14:43
  • $\begingroup$ I was mad about it. It's already fixed but will show only in 13.1 $\endgroup$ Commented Feb 1, 2022 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.