1
$\begingroup$

I am box filtering data from a DNS simulation. When filtering the original data read from a file, this is fast. When filtering a function of these data after applying Map, it is 10 times slower. If I write the data after Map operation to a file and read in again, the filtering is fast again.

What is the difference of data after Map operation to simple numbers? How can I convert the result to numbers without writing / reading to a file?

Here is a test example:


    idiff = 100; jdiff = 100; kdiff = 400;
    
    cfct3Dr = Array[0.5 &, {idiff + 1, jdiff + 1, kdiff + 1}];
    
    omcc[c_, m_] = -c^(1 + m) (-1 + c^m) (1 + m) // Simplify
    
    m0 = 49/11
    
    omm0[c_] = omcc[c, m0]

  
    omDNScalc = ParallelMap[omm0[#] &, cfct3Dr, {3}];
    
    Export["omDNSdata.h5", omDNScalc];
    omDNS = Import["omDNSdata.h5", "Dataset1"];
    Delta = 32;shift = 15;niLES =4;njLES=4;nkLES=24;

    For[omDNSLES = Array[0.*# &, {niLES, njLES, nkLES}]; i = 1, 
      i <= niLES, i++, 
      For[j = 1, j <= njLES, j++, 
       For[k = 1, k <= nkLES, k++, 
        omDNSLES[[i]][[j]][[k]] = 
         Total[Total[
            Total[omDNS[[shift*(i - 1) + 1 ;; shift*(i - 1) + Delta, 
              shift*(j - 1) + 1 ;; shift*(j - 1) + Delta, 
              shift*(k - 1) + 1 ;; 
               shift*(k - 1) + Delta]]]]]/(Delta^3)]]] // AbsoluteTiming
    {0.0406014, Null}
    
    For[omDNSLEScalc = Array[0.*# &, {niLES, njLES, nkLES}]; i = 1, 
      i <= niLES, i++, 
      For[j = 1, j <= njLES, j++, 
       For[k = 1, k <= nkLES, k++, 
        omDNSLEScalc[[i]][[j]][[k]] = 
         Total[Total[
            Total[omDNScalc[[shift*(i - 1) + 1 ;; shift*(i - 1) + Delta, 
               shift*(j - 1) + 1 ;; shift*(j - 1) + Delta, 
               shift*(k - 1) + 1 ;; 
                shift*(k - 1) + Delta]]]]]/(Delta^3)]]] // AbsoluteTiming
    {1.41565, Null}

In this example, niLES=4,njLES=4,nkLES=27,shift=12,Delta=32, cfct3Dr and cfomDNS are 3D (100x100x340) arrays, cfct3Dr is read in from a .h5 file. The second loop need 10 times more CPU time, I could not find a difference when inspecting omDNS and omDNScalc.

Also: Is there a more elegant way to box filter data from a bigger 3D array, maybe using Map instead of the nested For (or Do) lists?

$\endgroup$
3
  • 1
    $\begingroup$ Please make your code self-contained. There are several undefined variables. $\endgroup$ – Roman Dec 19 '20 at 16:20
  • 2
    $\begingroup$ omm0[c] is a syntax error. It should read: omm0[c_]. Therefore omDNScalc will not work as expected. $\endgroup$ – Daniel Huber Dec 19 '20 at 16:39
  • $\begingroup$ Your code does not run for me: i.stack.imgur.com/CgegL.png -- It looks like you're just constructing Table[] so that seems better that For[]. Your three nested Total[] seem like the same as Total[..., 3]. These are minor compared to Daniel's remark. $\endgroup$ – Michael E2 Dec 19 '20 at 18:48
3
$\begingroup$

The reason for the difference in speed is that omDNScalc is not a packed array. This is a faster way to compute it that constructs a packed array:

omDNScalc = omm0@cfct3Dr;

With that change, the times are comparable and fast.


Alternative: I had to experiment a little to figure out you don't sum over the whole array omDNS (that is, idiff etc. are not evenly covered by shift and Delta). That is the reason we don’t take all of omDNS below.

See Developer`PartitionMap for a useful function and Partition of which it is a generalization. The code is no faster (except for omm0@cfct3Dr), but it's shorter.

idiff = 100; jdiff = 100; kdiff = 400;
cfct3Dr = ConstantArray[0.5, {idiff + 1, jdiff + 1, kdiff + 1}];
omcc[c_, m_] = -c^(1 + m) (-1 + c^m) (1 + m) // Simplify;
m0 = 49/11;
omm0[c_] = omcc[c, m0];
omDNScalc = omm0@cfct3Dr;
omDNS = omDNScalc;
Delta = 32; shift = 15; niLES = 4; njLES = 4; nkLES = 24;

omDNSLES = Developer`PartitionMap[
    Total[#, Infinity] &,
    omDNS[[;; -shift, ;; -shift, ;; -shift]],
    {Delta, Delta, Delta},
    {shift, shift, shift}] / Delta^3;
$\endgroup$
4
  • $\begingroup$ Thanks, this helps my understanding a lot! $\endgroup$ – MichaelPfitzner Dec 19 '20 at 21:47
  • $\begingroup$ @MichaelPfitzner You're welcome. $\endgroup$ – Michael E2 Dec 19 '20 at 22:02
  • $\begingroup$ @MichaelPfitzner does this answer adequately address your question? If so, it is better for future users if you accept the answer by turning the checkmark green for this answer by clicking on it. $\endgroup$ – CA Trevillian Dec 20 '20 at 14:27
  • $\begingroup$ Thank you for the comment - I am a newby here and didn't know. $\endgroup$ – MichaelPfitzner Dec 21 '20 at 8:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.