8
$\begingroup$

Dealing with spectral data (2D lists) where offsets (corresponding 1D list) must be removed from each individual spectrum. Looking for feedback on improvements over my current method.

    offset = {53.0617, 52.5185, 53.2469, 52.8025, 53.2716, 53.284, 53.2716, 53.6049, 53.5062, 53.642};

    data = {{-0.0617284, -1.51852, -2.24691, -1.80247, -2.2716, -0.283951, -2.2716, -4.60494, -2.50617, -2.64198}, {0.938272, -1.51852, -0.246914, 0.197531, -0.271605, 0.716049, 1.7284, 0.395062, -0.506173, 1.35802}, {-0.0617284, 0.481481, -0.246914, 0.197531, -0.271605, -0.283951, -1.2716, -2.60494, -2.50617, -0.641975}, {-4.06173, -3.51852, -2.24691, -1.80247, -2.2716, -2.28395, -2.2716, -2.60494, -2.50617, -0.641975}, {-0.0617284, -1.51852, -2.24691, -1.80247, -2.2716, -2.28395, -2.2716, -2.60494, -2.50617, -4.64198}, {-2.06173, -1.51852, -0.246914, 0.197531, -2.2716, -2.28395, -1.2716, -0.604938, -0.506173, -2.64198}, {-4.06173, -3.51852, -2.24691, -3.80247, -4.2716, -3.28395, -2.2716, -3.60494, -4.50617, -4.64198}, {-3.06173, -3.51852, -2.24691, -3.80247, -3.2716, -2.28395, -2.2716, -3.60494, -4.50617, -2.64198}, {-2.06173, -3.51852, -2.24691, -3.80247, -3.2716, -2.28395, -2.2716, -2.60494, -2.50617, -4.64198}, {-1.06173, -3.51852, -2.24691, -1.80247, -4.2716, -3.28395, -1.2716, -0.604938, -0.506173, -0.641975}};

I currently always use the following MMA code snippet for processing.

    data = (# - offset) & /@ data;

Is there a better use of Thread, Map, etc. that may be considered? The data sets typically include 1000's of spectra each 1000-2000 points long. So 2D list with some million values.

$\endgroup$
  • $\begingroup$ What do you mean by "better"? More performant? $\endgroup$ – Carl Lange Dec 14 '19 at 20:03
  • $\begingroup$ In what sense better? Faster? $\endgroup$ – mikado Dec 14 '19 at 20:03
  • 1
    $\begingroup$ @mikado Jinx - we asked the same question within one second of the other! $\endgroup$ – Carl Lange Dec 14 '19 at 20:04
  • $\begingroup$ Faster is desired. Not concerned about memory usage, etc. $\endgroup$ – OpticsMan Dec 14 '19 at 20:16
  • $\begingroup$ Thanks for the suggestions. Never considered compiling such a simple expression. But the 5x speed-up is impressive. $\endgroup$ – OpticsMan Dec 15 '19 at 17:55
7
$\begingroup$
{n, m} = {10^4, 10^4};
offset = RandomReal[1, n];

data = RandomReal[1, {m, n}];

cf = Compile[{{v, _Real, 1}, {offset, _Real, 1}}, 
   Table[v[[i]] - offset[[i]], {i, Length[v]}],
   RuntimeAttributes -> {Listable}, CompilationTarget -> "C", 
   RuntimeOptions -> "Speed"];

r1 = (# - offset) & /@ data; // RepeatedTiming
r2 = Plus[data, ConstantArray[-offset, m]]; // RepeatedTiming
r3 = ArrayReshape[Outer[Plus, Developer`ToPackedArray@{-offset}, data, 1], 
      {m, n}]; // RepeatedTiming
r4 = cf[data, offset]; // RepeatedTiming

r1 == r2 == r3 == r4

Output

{1.08, Null}

{0.557, Null}

{0.233, Null}

{0.20, Null}

True

$\endgroup$
6
$\begingroup$

A slightly faster method uses KroneckerProduct to create a suitable matrix of offsets. Some data:

{n, m} = {10^4, 10^4};
offset = RandomReal[100, n];
data = RandomReal[100, {m, n}];

Your method:

r1 = (#-offset)& /@ data; //AbsoluteTiming

{1.80568, Null}

Using KroneckerProduct:

r2 = data + KroneckerProduct[ConstantArray[-1., m], offset]; //AbsoluteTiming

{0.830738, Null}

Check:

r1 == r2

True

$\endgroup$
  • 3
    $\begingroup$ While I love a good use of KroneckerProduct, why not just use ConstantArray[-offset, m]? $\endgroup$ – That Gravity Guy Dec 14 '19 at 22:29
4
$\begingroup$

The Map version looks efficient compared with MapThread.

data2 = Flatten[ConstantArray[data, 100000], 1];
First[Timing[data3 = (# - offset) & /@ data2;]]

0.384383

First[Timing[
  data4 = MapThread[
     Plus, {data2, -ConstantArray[offset, Length[data2]]}];]]

2.80672

data3 == data4

True

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.