3
$\begingroup$

I have a 900x100x100 array of floats called 'data' which is essentially 900 frames of 100x100 pixels. I want to extract 10,000 sequences of 900 values each. These are essentially the time series of the values of each the 10,000 individual pixels.

I have achieved this in what I know is an ugly and horribly inefficient iterative way by doing the following

output = {}
reshapedTensor = Table[data[[;;,i,j]],{i,1,100},{j,1,100}]
Do[AppendTo[output,reshapedTensor[[i,j]]],{i,100},{j,100}]]

where I am essentially walking across the 100x100 frame and pulling out each 900 point sequence and then appending it to the list output.

Yes horrendous, I know, but I found the concept of "levels" in some of the ArrayReshape/Flatten functions confusing and couldn't tell how to apply them here.

I would like to both find the correct functional way to do this and also understand the native functions that reorder data in this sort of way so I can use these better in future.

I suspect it will also be much more performant and easier to read and make sense of.

Thanks much!

$\endgroup$
2
  • 3
    $\begingroup$ Does Flatten[data, {{2, 3}, {1}}] do what you want? $\endgroup$
    – Carl Woll
    Nov 7 '21 at 5:33
  • $\begingroup$ I don't know, I will try it, but I would like to understand what this means. Does {2,3} mean iterate over the 2nd and 3rd dimension and {1} mean pick the 1st dimension as a value? This is the part I was confused about. So any explanation of the meaning would be helpful. Thanks much. $\endgroup$
    – Nitin
    Nov 7 '21 at 5:38
3
$\begingroup$

Flatten seems convenient, but Transpose and ArrayReshape should be faster for packed arrays:

m = 900;
n = 100;
A = RandomReal[{-1, 1}, {m, n, n}];

B1 = Flatten[A, {{2, 3}, {1}}]; // RepeatedTiming // First
B2 = ArrayReshape[Transpose[A, {3, 1, 2}], {n n, m}]; // RepeatedTiming // First

0.15

0.042

True

For an array R of dimension {k,m,n}, S = Transpose[A, {3, 1, 2}] swaps the dimensions such that S has dimensions {m,n,k}. Read {3, 1, 2} as "slot one moves to place 3, slot 2 moves to slot 1, slot 3 moves to slot 2".

ArrayReshape takes a given array and just reinterprets it as an array of the requested dimensions -- at least if the total number of elements of the input and the desired output coincide. (It has to do some padding otherwise.) In this case, only the vector of dimensions has to be changed; the data can stay where it is. Well, Mathematica might or might not have to make a copy. There is a good chance that a copied is not made in the code for B2 above, because it could in principle exploit that the return value of Transpose is a temporary object and thus can be modified in-place when handed to ArrayReshape. The following timing experiment confirms that Mathematica is indeed clever enough:

B2 = ArrayReshape[Transpose[A, {3, 1, 2}], {n n, m}]; // 
  RepeatedTiming // First
B3 = Transpose[A, {3, 1, 2}]; // RepeatedTiming // First
(B4 = A; B4[[1, 1, 1]] = 0.; ) // RepeatedTiming // First

0.039

0.040

0.015

This shows that ArrayReshape does not make any measurable difference compared to the cost of a plain copy (third timing). In case you wonder why I modified B4 after copying: I had to do so in order to enforce a physical copy because Mathematica uses lazy copying. This means that the following just sets a pointer and does not perform any copy at all:

(B4 = A;) // RepeatedTiming // First

3.*10^-7

$\endgroup$
2
  • $\begingroup$ Thank you so much!! Very well explained, in terms of both what will work and why it works and also how to use it in future. The internal behavior of Mathematica re arrays in memory was especially helpful! $\endgroup$
    – Nitin
    Nov 7 '21 at 17:23
  • $\begingroup$ Pleased to hear that. You're welcome! $\endgroup$ Nov 7 '21 at 17:25
3
$\begingroup$

Here's an example array:

data = Array[a, {4,2,2}]

{{{a[1, 1, 1], a[1, 1, 2]}, {a[1, 2, 1], a[1, 2, 2]}}, {{a[2, 1, 1], a[2, 1, 2]}, {a[2, 2, 1], a[2, 2, 2]}}, {{a[3, 1, 1], a[3, 1, 2]}, {a[3, 2, 1], a[3, 2, 2]}}, {{a[4, 1, 1], a[4, 1, 2]}, {a[4, 2, 1], a[4, 2, 2]}}}

Use Flatten:

Flatten[data, {{2,3}, {1}}]

{{a[1, 1, 1], a[2, 1, 1], a[3, 1, 1], a[4, 1, 1]}, {a[1, 1, 2], a[2, 1, 2], a[3, 1, 2], a[4, 1, 2]}, {a[1, 2, 1], a[2, 2, 1], a[3, 2, 1], a[4, 2, 1]}, {a[1, 2, 2], a[2, 2, 2], a[3, 2, 2], a[4, 2, 2]}}

Compare to your code:

output={};
reshapedTensor=Table[data[[;;,i,j]],{i,1,2},{j,1,2}];
Do[AppendTo[output, reshapedTensor[[i,j]]],{i,2},{j,2}]
output

{{a[1, 1, 1], a[2, 1, 1], a[3, 1, 1], a[4, 1, 1]}, {a[1, 1, 2], a[2, 1, 2], a[3, 1, 2], a[4, 1, 2]}, {a[1, 2, 1], a[2, 2, 1], a[3, 2, 1], a[4, 2, 1]}, {a[1, 2, 2], a[2, 2, 2], a[3, 2, 2], a[4, 2, 2]}}

$\endgroup$
2
  • $\begingroup$ Yes, I tried it and it gives me what I want. So I have the functional expression that works. I am afraid the above example still hasn't clarified much for me in how I would use this in other contexts. But I will read the fine manual again and try to make sense of this. Thanks very much for the help. $\endgroup$
    – Nitin
    Nov 7 '21 at 6:01
  • $\begingroup$ The example from the Flatten help page which goes Flatten[RandomReal[1, {3, 5, 7}], {{2, 3}, {1}}] // Dimensions was very helpful along with your initial comment so thanks for that!! $\endgroup$
    – Nitin
    Nov 7 '21 at 6:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.