How to reorder a 3d (m x n x n) array to extract an n**2 length list of sequences of m points

Question

I have a 900x100x100 array of floats called 'data' which is essentially 900 frames of 100x100 pixels. I want to extract 10,000 sequences of 900 values each. These are essentially the time series of the values of each the 10,000 individual pixels.

I have achieved this in what I know is an ugly and horribly inefficient iterative way by doing the following

output = {}
reshapedTensor = Table[data[[;;,i,j]],{i,1,100},{j,1,100}]
Do[AppendTo[output,reshapedTensor[[i,j]]],{i,100},{j,100}]]

where I am essentially walking across the 100x100 frame and pulling out each 900 point sequence and then appending it to the list output.

Yes horrendous, I know, but I found the concept of "levels" in some of the ArrayReshape/Flatten functions confusing and couldn't tell how to apply them here.

I would like to both find the correct functional way to do this and also understand the native functions that reorder data in this sort of way so I can use these better in future.

I suspect it will also be much more performant and easier to read and make sense of.

Thanks much!

Answer 1

Flatten seems convenient, but Transpose and ArrayReshape should be faster for packed arrays:

m = 900;
n = 100;
A = RandomReal[{-1, 1}, {m, n, n}];

B1 = Flatten[A, {{2, 3}, {1}}]; // RepeatedTiming // First
B2 = ArrayReshape[Transpose[A, {3, 1, 2}], {n n, m}]; // RepeatedTiming // First

0.15

0.042

True

For an array R of dimension {k,m,n}, S = Transpose[A, {3, 1, 2}] swaps the dimensions such that S has dimensions {m,n,k}. Read {3, 1, 2} as "slot one moves to place 3, slot 2 moves to slot 1, slot 3 moves to slot 2".

ArrayReshape takes a given array and just reinterprets it as an array of the requested dimensions -- at least if the total number of elements of the input and the desired output coincide. (It has to do some padding otherwise.) In this case, only the vector of dimensions has to be changed; the data can stay where it is. Well, Mathematica might or might not have to make a copy. There is a good chance that a copied is not made in the code for B2 above, because it could in principle exploit that the return value of Transpose is a temporary object and thus can be modified in-place when handed to ArrayReshape. The following timing experiment confirms that Mathematica is indeed clever enough:

B2 = ArrayReshape[Transpose[A, {3, 1, 2}], {n n, m}]; // 
  RepeatedTiming // First
B3 = Transpose[A, {3, 1, 2}]; // RepeatedTiming // First
(B4 = A; B4[[1, 1, 1]] = 0.; ) // RepeatedTiming // First

0.039

0.040

0.015

This shows that ArrayReshape does not make any measurable difference compared to the cost of a plain copy (third timing). In case you wonder why I modified B4 after copying: I had to do so in order to enforce a physical copy because Mathematica uses lazy copying. This means that the following just sets a pointer and does not perform any copy at all:

(B4 = A;) // RepeatedTiming // First

3.*10^-7

Answer 2

Here's an example array:

data = Array[a, {4,2,2}]

{{{a[1, 1, 1], a[1, 1, 2]}, {a[1, 2, 1], a[1, 2, 2]}}, {{a[2, 1, 1], a[2, 1, 2]}, {a[2, 2, 1], a[2, 2, 2]}}, {{a[3, 1, 1], a[3, 1, 2]}, {a[3, 2, 1], a[3, 2, 2]}}, {{a[4, 1, 1], a[4, 1, 2]}, {a[4, 2, 1], a[4, 2, 2]}}}

Use Flatten:

Flatten[data, {{2,3}, {1}}]

{{a[1, 1, 1], a[2, 1, 1], a[3, 1, 1], a[4, 1, 1]}, {a[1, 1, 2], a[2, 1, 2], a[3, 1, 2], a[4, 1, 2]}, {a[1, 2, 1], a[2, 2, 1], a[3, 2, 1], a[4, 2, 1]}, {a[1, 2, 2], a[2, 2, 2], a[3, 2, 2], a[4, 2, 2]}}

Compare to your code:

output={};
reshapedTensor=Table[data[[;;,i,j]],{i,1,2},{j,1,2}];
Do[AppendTo[output, reshapedTensor[[i,j]]],{i,2},{j,2}]
output

{{a[1, 1, 1], a[2, 1, 1], a[3, 1, 1], a[4, 1, 1]}, {a[1, 1, 2], a[2, 1, 2], a[3, 1, 2], a[4, 1, 2]}, {a[1, 2, 1], a[2, 2, 1], a[3, 2, 1], a[4, 2, 1]}, {a[1, 2, 2], a[2, 2, 2], a[3, 2, 2], a[4, 2, 2]}}