Flatten command: matrix as second argument

Question

One thing I could never wrap my head around is how Flatten works when provided with a matrix as the second argument, and the Mathematica help isn't particularly good on this one.

Taken from the Flatten Mathematica documentation:

Flatten[list, {{s11, s12, ...}, {s21, s22, ...}, ...}]

Flattens list by combining all levels $s_{ij}$ to make each level $i$ in the result.

Could someone elaborate on what this actually means/does?

Answer 1

One convenient way to think of Flatten with the second argument is that it performs something like Transpose for ragged (irregular) lists. Here is a simple example:

In[63]:=  Flatten[{{1,2,3},{4,5},{6,7},{8,9,10}},{{2},{1}}]
Out[63]= {{1,4,6,8},{2,5,7,9},{3,10}}

What happens is that elements which constituted level 1 in the original list are now constituents at level 2 in the result, and vice versa. This is exactly what Transpose does, but done for irregular lists. Note however, that some information about positions is lost here, so we can not directly inverse the operation:

In[65]:= Flatten[{{1,4,6,8},{2,5,7,9},{3,10}},{{2},{1}}]
Out[65]= {{1,2,3},{4,5,10},{6,7},{8,9}}

To have it reversed correctly, we'd have to do something like this:

In[67]:= Flatten/@Flatten[{{1,4,6,8},{2,5,7,9},{3,{},{},10}},{{2},{1}}]
Out[67]= {{1,2,3},{4,5},{6,7},{8,9,10}}

A more interesting example is when we have deeper nesting:

In[68]:= Flatten[{{{1,2,3},{4,5}},{{6,7},{8,9,10}}},{{2},{1},{3}}]
Out[68]= {{{1,2,3},{6,7}},{{4,5},{8,9,10}}}

Here again, we can see that Flatten effectively worked like (generalized) Transpose, interchanging pieces at the first 2 levels. The following will be harder to understand:

In[69]:=  Flatten[{{{1, 2, 3}, {4, 5}}, {{6, 7}, {8, 9,  10}}}, {{3}, {1}, {2}}]
Out[69]= {{{1, 4}, {6, 8}}, {{2, 5}, {7, 9}}, {{3}, {10}}}

The following image illustrates this generalized transpose:

We may do it in two consecutive steps:

In[72]:=  step1 = Flatten[{{{1,2,3},{4,5}},{{6,7},{8,9,10}}},{{1},{3},{2}}]
Out[72]= {{{1,4},{2,5},{3}},{{6,8},{7,9},{10}}}

In[73]:= step2 =  Flatten[step1,{{2},{1},{3}}]
Out[73]= {{{1,4},{6,8}},{{2,5},{7,9}},{{3},{10}}}

Since the permutation {3,1,2} can be obtained as {1,3,2} followed by {2,1,3}. Another way to see how it works is to use numbers which indicate the position in the list structure:

Flatten[{{{111, 112, 113}, {121, 122}}, {{211, 212}, {221, 222, 223}}}, {{3}, {1}, {2}}]
(*
==> {{{111, 121}, {211, 221}}, {{112, 122}, {212, 222}}, {{113}, {223}}}
*)

From this, one can see that in the outermost list (first level), the third index (corresponding the third level of the original list) grows, in each member list (second level) the first element grows per element (corresponding to the first level of the original list), and finally in the innermost (third level) lists, the second index grows, corresponding to the second level in the original list. Generally, if the k-th element of the list passed as second element is {n}, growing the k-th index in the resulting list structure corresponds to increasing the n-th index in the original structure.

Finally, one can combine several levels to effectively flatten the sub-levels, like so:

In[74]:=  Flatten[{{{1,2,3},{4,5}},{{6,7},{8,9,10}}},{{2},{1,3}}]
Out[74]= {{1,2,3,6,7},{4,5,8,9,10}}

Answer 2

A second list argument to Flatten serves two purposes. First, it specifies the order in which indices will be iterated when gathering elements. Second, it describes list flattening in the final result. Let's look at each of these capabilities in turn.

Iteration Order

Consider the following matrix:

$m = Array[Subscript[m, Row[{##}]]&, {4, 3, 2}];
$m // MatrixForm

We can use a Table expression to create a copy of the matrix by iterating over all of its elements:

$m === Table[$m[[i, j, k]], {i, 1, 4}, {j, 1, 3}, {k, 1, 2}]
(* True *)

This identity operation is uninteresting, but we can transform the array by swapping the order of the iteration variables. For example, we can swap i and j iterators. This amounts to swapping the level 1 and level 2 indices and their corresponding elements:

$r = Table[$m[[i, j, k]], {j, 1, 3}, {i, 1, 4}, {k, 1, 2}];
$r // MatrixForm

If we look carefully, we can see that each original element $m[[i, j, k]] will be found to correspond to the resulting element $r[[j, i, k]] -- the first two indices have been "swapped".

Flatten allows us to express an equivalent operation to this Table expression more succintly:

$r === Flatten[$m, {{2}, {1}, {3}}]
(* True *)

The second argument of the Flatten expression explicitly specifies the desired index order: indices 1, 2, 3 are altered to become indices 2, 1, 3. Note how we did not need to specify a range for each dimension of the array -- a significant notational convenience.

The following Flatten is an identity operation since it specifies no change to index order:

$m === Flatten[$m, {{1}, {2}, {3}}]
(* True *)

Whereas the following expression re-arranges all three indices: 1, 2, 3 -> 3, 2, 1

Flatten[$m, {{3}, {2}, {1}}] // MatrixForm

Again, we can verify that an original element found at the index [[i, j, k]] will now be found at [[k, j, i]] in the result.

If any indices are omitted from a Flatten expression, they are treated as if they had been specified last and in their natural order:

Flatten[$m, {{3}}] === Flatten[$m, {{3}, {1}, {2}}]
(* True *)

This last example can be abbreviated even further:

Flatten[$m, {3}] === Flatten[$m, {{3}}]
(* True *)

An empty index list results in the identity operation:

$m === Flatten[$m, {}] === Flatten[$m, {1}] === Flatten[$m, {{1}, {2}, {3}}]
(* True *)

That takes care of iteration order and index swapping. Now, let's look at...

List Flattening

One might wonder why we had to specify each index in a sublist in the previous examples. The reason is that each sublist in the index specification specifies which indices are to be flattened together in the result. Consider again the following identity operation:

Flatten[$m, {{1}, {2}, {3}}] // MatrixForm

What happens if we combine the first two indices into the same sublist?

Flatten[$m, {{1, 2}, {3}}] // MatrixForm

We can see that the original result was a 4 x 3 grid of pairs, but the second result is a simple list of pairs. The deepest structure, the pairs, were left untouched. The first two levels have been flattened into a single level. The pairs in the third level of the source matrix remained unflattened.

We could combine the second two indices instead:

Flatten[$m, {{1}, {2, 3}}] // MatrixForm

This result has the same number of rows as the original matrix, meaning that the first level was left untouched. But each result row has a flat list of six elements taken from the corresponding original row of three pairs. Thus, the lower two levels have been flattened.

We can also combine all three indices to get a completely flattened result:

Flatten[$m, {{1, 2, 3}}]

This can be abbreviated:

Flatten[$m, {{1, 2, 3}}] === Flatten[$m, {1, 2, 3}] === Flatten[$m]
(* True *)

Flatten also offers a shorthand notation when no index swapping is to take place:

$n = Array[n[##]&, {2, 2, 2, 2, 2}];

Flatten[$n, {{1}, {2}, {3}, {4}, {5}}] === Flatten[$n, 0]
(* True *)

Flatten[$n, {{1, 2}, {3}, {4}, {5}}] === Flatten[$n, 1]
(* True *)

Flatten[$n, {{1, 2, 3}, {4}, {5}}] === Flatten[$n, 2]
(* True *)

Flatten[$n, {{1, 2, 3, 4}, {5}}] === Flatten[$n, 3]
(* True *)

"Ragged" Arrays

All of the examples so far have used matrices of various dimensions. Flatten offers a very powerful feature that makes it more than just an abbreviation for a Table expression. Flatten will gracefully handle the case where sublists at any given level have differing lengths. Missing elements will be quietly ignored. For example, a triangular array can be flipped:

$t = Array[# Range[#]&, {5}];
$t // TableForm
(*
1               
2   4           
3   6   9       
4   8   12  16  
5   10  15  20  25
*)

Flatten[$t, {{2}, {1}}] // TableForm
(*
1   2   3   4   5
4   6   8   10  
9   12  15      
16  20          
25              
*)

... or flipped and flattened:

Flatten[$t, {{2, 1}}]
(* {1,2,3,4,5,4,6,8,10,9,12,15,16,20,25} *)

Answer 3

I learned a lot from WReach's and Leonid's answers and I'd like to make a small contribution:

It seems worth emphasizing that the primary intention of the list-valued second argument of Flatten is merely to flatten certain levels of lists (as WReach mentions in his List Flattening section). Using Flatten as a ragged Transpose seems like a side-effect of this primary design, in my opinion.

For example, yesterday I needed to transform this list

lists = {
   {{{1, 0}, {1, 1}}, {{2, 0}, {2, 4}}, {{3, 0}}},
   {{{1, 2}, {1, 3}}, {{2, Sqrt[2]}}, {{3, 4}}} 
   (*, more lists... *)
   };

into this one:

list2 = {
  {{1, 0}, {1, 1}, {2, 0}, {2, 4}, {3, 0}},
  {{1, 2}, {1, 3}, {2, Sqrt[2]}, {3, 4}}
  (*, more lists... *)
  }

That is, I needed to crush the 2nd and 3rd list-levels together.

I did it with

list2 = Flatten[lists, {{1}, {2, 3}}];

Answer 4

This is a old question, but frequently asked by a lot of people. Today when I was trying to explain how this works, I came across a quite clear explanation, so I think sharing it here would be helpful for further audience.

What do index means?

First let's make clear what index is: In Mathematica every expression is a tree, for example, let's look at a list:

TreeForm@{{1,2},{3,4}}

How you navigate in a tree?

Simple! You start from the root and at each crossing choose which way to go, for example, here if you want to reach 2, you begin with choosing the first path, then choose the second path. Let's write it out as {1,2} which is the just the index of element 2 in this expression.

How to understand Flatten?

Here consider a simple question, if I don't provide you with a complete expression, but instead I give you all the elements and their indexes, how you construct the original expression? For example, here I give you:

{<|"index" -> {1, 1}, "value" -> 1|>, <|"index" -> {1, 2}, "value" -> 2|>, <|"index" -> {2, 1}, "value" -> 3|>, <|"index" -> {2, 2}, "value" -> 4|>}

and tell you all heads are List, so what's the original expression?

Well, surely you can reconstruct the original expression as {{1,2},{3,4}}, but how? You probably can list the following steps:

1. First we look at the first element of index and sort and gather by it. Then we know that the first element of the whole expression should contain the first two elements in the original list...
2. Then we continue to look at the second argument, do the same...
3. Finally we get the original list as {{1,2},{3,4}}.

Well, that's reasonable! So what if I tell you, no, you should first sort and gather by the second element of the index and then gather by the first element of the index? Or I say we don't gather them twice, we just sort by both elements but give the first argument higher priority?

Well, you would probably get the following two list respectively, right?

1. {{1,3},{2,4}}
2. {1,2,3,4}

Well, check by yourself, Flatten[{{1,2},{3,4}},{{2},{1}}] and Flatten[{{1,2},{3,4}},{{1,2}}] do the same!

So, how you understand the second argument of Flatten?

1. Each list element inside the main list, for example, {1,2}, means you should GATHER all the lists according to these elements in the index, in other words these levels.
2. The order inside a list element represents how you SORT the elements gathered inside a list in previous step. for example, {2,1} means the position at the second level has higher priority than the position at the first level.

Examples

Now let's have some practice to be familiar with previous rules.

1. Transpose

The goal of Transpose on a simple m*n matrix is to make $A_{i,j} \rightarrow A^T_{j,i}$. But we can consider it another way, originally we sort the element by their i index first then sort them by their j index, now all we need to do is to change to sort them by j index first then by i next! So the code becomes:

Flatten[mat,{{2},{1}}]

Simple, right?

2. Traditional Flatten

The goal of traditional flatten on a simple m*n matrix is to create a 1D array instead of a 2D matrix, for example: Flatten[{{1,2},{3,4}}] returns {1,2,3,4}. This means that we don't gather elements this time, we only sort them, first by their first index then by the second:

Flatten[mat,{{1,2}}]

3. ArrayFlatten

Let's discuss a most simple case of ArrayFlatten, here we have a 4D list:

{{{{1,2},{5,6}},{{3,4},{7,8}}},{{{9,10},{13,14}},{{11,12},{15,16}}}}

so how we can do such a conversion to make it a 2D list?

$\left( \begin{array}{cc} \left( \begin{array}{cc} 1 & 2 \\ 5 & 6 \\ \end{array} \right) & \left( \begin{array}{cc} 3 & 4 \\ 7 & 8 \\ \end{array} \right) \\ \left( \begin{array}{cc} 9 & 10 \\ 13 & 14 \\ \end{array} \right) & \left( \begin{array}{cc} 11 & 12 \\ 15 & 16 \\ \end{array} \right) \\ \end{array} \right) \rightarrow \left( \begin{array}{cccc} 1 & 2 & 3 & 4 \\ 5 & 6 & 7 & 8 \\ 9 & 10 & 11 & 12 \\ 13 & 14 & 15 & 16 \\ \end{array} \right)$

Well, this is simple too, we need the group by the original first and the third level index first, and we should give the first index higher priority in sorting. The same goes to the second and the forth level:

Flatten[mat,{{1,3},{2,4}}]

4. "Resize" a image

Now we have a image, for example:

img=Image@RandomReal[1,{10,10}]

But it's definitely too small for us to view, so we want to make it bigger by extending each pixel to a 10*10 size huge pixel.

First we shall try:

ConstantArray[ImageData@img,{10,10}]

But it returns a 4D matrix with dimensions {10,10,10,10}. So we should Flatten it. This time we want the third argument to takes higher priority instead of the first one, so a minor tuning would work:

Image@Flatten[ConstantArray[ImageData@img,{10,10}],{{3,1},{4,2}}]

A comparison:

Hope this could help!