# Is Mathematica's implementation of convolutional neural networks really inefficient?

I'm surprised how slow ConvolutionLayer works with TargetDevice->"CPU". For instance this code:

data = RandomReal[{-.1, .1}, {1, 1024, 1024}];
net = NetInitialize@
NetChain[{ConvolutionLayer[1, 3], Ramp},
"Input" -> Dimensions[data]];
net[data, TargetDevice -> "CPU"]; // RepeatedTiming


calculates a simple convolution of a 1024x1024 buffer with a 3x3 filter. This takes 36ms on my PC, using 8 cores. This seems very slow.

For comparison: If I use low-level IPP functions to calculate a convolution of a 1024x1024 buffer (64bit floating point data type) with a 3x3 filter, it takes about 1.2ms, using only a single core.

Is the mxnet CPU convolution implementation really 250x less efficient than IPP?

Clarification: The IPP (Intel Performance Primitives) is a low-level CPU library for signal-processing and image processing tasks, optimized for Intel CPUs. So I'm comparing two different CPU implementations.

A completely naive C implementation (below) takes about 3ms for a 1024x1024 * 3x3 convolution. Isn't 'ConvolutionLayer[1,3]' doing more or less the same? Why does it take 10 times as long using 8 times as many cores?

for (int y=1;y<h-1; ++y)
for (int x = 1; x < w-1; ++x)
{
double s = 0;
for (int dy = -1; dy <= 1; ++dy)
for (int dx = -1; dx <= 1; ++dx)
s += src[(y + dy)*w + x + dx] * flt[dx + 1 + 3 * (dy + 1)];
dst[y*w + x] = s;
}

• From what you say in your question, it seems the answer is "yes" 😉 – Carl Lange May 7 at 16:32
• Yep. The GPU market is booming right now, especially with "external GPUs" precisely for applications like this. You'll find a lot on the topic (and the reasons why GPUs excel in these scenarios) in a quick Google search. – user6014 May 7 at 17:11
• @user6014: I'm not sure I follow. What does this have to do with GPUs? IPP is a CPU-only library. It seems as if the mxnet CPU implementation is simply inefficient as hell, and could be >200 times faster on the CPU if it was optimized well. – Niki Estner May 7 at 19:14

Exploration of the problem. This is not an answer.

SeedRandom[0];
img = RandomImage[1, {1024, 1024}, ColorSpace -> "Grayscale"];

enc = NetEncoder[{"Image", {1024, 1024}, ColorSpace -> "Grayscale"}];

eimg = enc@img;

conv = ConvolutionLayer[1, 3, "Input" -> {1, 1024, 1024}] // NetInitialize;

conv@eimg; // RepeatedTiming


{0.0428, Null}

w = NetExtract[conv, "Weights"][[1, 1]] // Normal;

ImageConvolve[img, Reverse /@ Reverse@w, Padding -> None]; // RepeatedTiming


{0.0055, Null}

(conv@eimg)[[1, 1, ;; 10]]


{-1.27131,-1.58433,-2.09876,-1.88886,-1.69361,-2.02711,-1.13003,-0.975968,-1.394,-1.95289}

ImageConvolve[img, Reverse /@ Reverse@w, Padding -> None] // ImageData // #[[1, ;; 10]] &


{-1.27131,-1.58433,-2.09876,-1.88886,-1.69361,-2.02711,-1.13003,-0.975968,-1.394,-1.95289}