# Extract data from ragged arrays

I have data in ragged arrays of the form

$$\left( \begin{array}{cc} \text{x1} & \{\text{a1}\} \\ \text{x2} & \{\text{a2}\} \\ \text{x3} & \{\text{a3},\text{b3},\text{c3}\} \\ \text{x4} & \{\text{a4},\text{b4},\text{c4}\} \\ \text{x5} & \{\text{a5},\text{b5},\text{c5},\text{d5},\text{e5}\} \\ \text{x6} & \{\text{a6}\} \\ \text{x7} & \{\text{a7},\text{b7}\} \\ \text{x8} & \{\text{a8},\text{b8},\text{c8}\} \\ \text{x9} & \{\text{a9}\} \\ \end{array} \right)$$

and I wish to sort it into arrays such as $$\text{data1}=\left( \begin{array}{cc} \text{x1} & \text{a1} \\ \text{x2} & \text{a2} \\ \text{x3} & \text{a3} \\ \text{x4} & \text{a4} \\ \text{x5} & \text{a5} \\ \text{x6} & \text{a6} \\ \text{x7} & \text{a7} \\ \text{x8} & \text{a8} \\ \text{x9} & \text{a9} \\ \end{array} \right);$$

and $$\text{data2}=\left( \begin{array}{cc} \text{x3} & \text{b3} \\ \text{x4} & \text{b4} \\ \text{x5} & \text{b5} \\ \text{x7} & \text{b7} \\ \text{x8} & \text{b8} \\ \end{array} \right);$$

and $$\text{data3}=\left( \begin{array}{cc} \text{x3} & \text{c3} \\ \text{x4} & \text{c4} \\ \text{x5} & \text{c5} \\ \text{x8} & \text{c8} \\ \end{array} \right);$$

finally $$\text{data4}=\left( \begin{array}{cc} \text{x5} & \text{e5} \\ \end{array} \right);$$

The actual data is numeric and has many more rows and an unknown number of ragged columns. I have been trying with Cases but with no success so far. Any ideas? Thanks

***** Edit *****

Thank you all for your efforts. There is much to learn from.

I have had a go at timing the individual offerings.

ClearAll[kglr, u1066, syed1, syed2, lericr, eldo]

nn = 5000 {3, 4, 5, 6, 7};
data = Flatten[
Table[{#[[1]], Rest[#]} & /@
RandomReal[{-1, 1}, {nn[[i]], i + 1}], {i, Length@nn}], 1];

(*    *)
tu1066 =
Timing[Through[(Cases[{x_, #} :> {x, y}] & /@
NestList[Prepend[_], {y_, ___}, 3])[data]];;][[1]];
(*    *)
tsyed1 = Timing[Flatten[Thread /@ data, {{2}, {1}}];][[1]];
tsyed2 = Timing[tlist = Thread /@ data;
Clear[f];
f[k_List, n_Integer] :=
Scan[If[Length@# > n - 1, Sow[Part[#, n]], Nothing] &, k] //
Reap // Last // First;
f[tlist, #] & /@ Range[Max@(Length /@ tlist)];][[1]];
(*    *)
tlericr =
Timing[Flatten[
List], {2}];][[1]];
(*    *)
teldo = Timing[data1 = <|Rule @@@ data|>;
get[n_] := KeyValueMap[List]@DeleteMissing@Query[All, n]@data1;
{get[1], get[2], get[3], get[4], get[5]};][[1]];
(*    *)


The results are

TableForm[{tkglr, tu1066, tsyed1, tsyed2, tlericr, teldo},
TableHeadings -> {{kglr, u1066, syed1, syed2, lericr, eldo}, None}]


$$\begin{array}{c|c} \text{kglr} & 0.140625 \\ \text{u1066} & 0.15625 \\ \text{syed1} & 0.140625 \\ \text{syed2} & 1.01563 \\ \text{lericr} & 0.171875 \\ \text{eldo} & 2.45313 \\ \end{array}$$

So the methods of kglr and the first method of syed are the fastest.

Well done everyone

• Shouldn't there also be a {x5, d5}? Commented Sep 28, 2023 at 15:41
• Yes, in my opinion
– eldo
Commented Sep 28, 2023 at 15:42

raggedThread = Flatten[Map[Thread] @ #, {2}]&;


{{{x1, a1}, {x2, a2}, {x3, a3}, {x4, a4}, {x5, a5},
{x6, a6}, {x7, a7}, {x8, a8}, {x9, a9}},
{{x3, b3}, {x4, b4}, {x5, b5}, {x7, b7}, {x8, b8}},
{{x3, c3}, {x4, c4}, {x5, c5}, {x8, c8}},
{{x5, d5}},
{{x5, e5}}}

Grid[MapThread[Prepend] @ {pairlists, "D" <> ToString[#] & /@ Range[5]},
Dividers -> {{False, 2 -> True}, False}]


This also works:

raggedThread2 = Apply[Join[##, 2] &] @* Map[Map[List]] @* Map[Thread];

• Ah, there it is. Map[Thread][...] instead of MapThread[...] (which is what I did). Commented Sep 28, 2023 at 21:44
• @kglr Your method is the fastest. Thanks
– Hugh
Commented Oct 5, 2023 at 14:09

With @eldo's data:

list = {{x1, {a1}}, {x2, {a2}}, {x3, {a3, b3, c3}}, {x4, {a4, b4,
c4}}, {x5, {a5, b5, c5, d5, e5}}, {x6, {a6}}, {x7, {a7,
b7}}, {x8, {a8, b8, c8}}, {x9, {a9}}};


Using Flatten:

Since Flatten can transpose ragged arrays:

Flatten[Thread /@ list, {{2}, {1}}] // Grid


Using Sow/Reap:

tlist= Thread/@list;

Clear[f];
f[k_List, n_Integer] :=
Scan[If[Length@# > n - 1, Sow[Part[#, n]], Nothing] &, k] // Reap //
Last // First

f[tlist, #] & /@ Range[Max@(Length /@ tlist)] // Grid


Result

• Your first method is the fastest. Equal to kglr. Thanks
– Hugh
Commented Oct 5, 2023 at 14:11

I think something like this might work:

(* This is my test data, since you didn't provide yours in copy-pasteable form, and I was too lazy to reproduce by hand. I did include an empty list in the ragged data as a robustness test case. *)
xs = {x1, x2, x3, x4};
data = {{a1}, {a2, b2, c2}, {}, {a4, b4}};

(*
{{{x1,a1},{x2,a2},{x4,a4}},{{x2,b2},{x4,b4}},{{x2,c2}}}
*)


Not sure whether you want this form or if you want the transform of each sub-structure. You can map Transform over this if you want.

Alt

One character shorter, and might be clearer semantically...or not?

Flatten[Inner[Thread@*List, xs, data, List], {2}]


Old

Partition[Flatten[MapThread[Thread@*List, {xs, data}], {2, 3}], 2]

(*
{{{x1,x2,x4},{a1,a2,a4}},{{x2,x4},{b2,b4}},{{x2},{c2}}}
*)


This feels a bit hack-ish, and it seems like there should be a more elegant way.

Each of your dataN can be extracted from this.

• Thanks. Not sure why this is slower than kglr's. Seems similar.
– Hugh
Commented Oct 5, 2023 at 14:14
data1 = Cases[list, {x_,{y_,___}}:>{x,y}]

(* {{x1,a1},{x2,a2},{x3,a3},{x4,a4},{x5,a5},{x6,a6},{x7,a7},{x8,a8},{x9,a9}} *)

data2 = Cases[list, {x_,{_,y_,___}}:>{x,y}]

(* {{x3,b3},{x4,b4},{x5,b5},{x7,b7},{x8,b8}} *)

data3 = Cases[list, {x_,{_,_,y_,___}}:>{x,y}]

(* {{x3,c3},{x4,c4},{x5,c5},{x8,c8}} *)

data4= Cases[list, {x_,{_,_,_,y_,___}}:>{x,y}]

(* {{x5,d5}}  *)



Or

{data1x,data2x,data3x,data4x} = Through[(Cases[{x_,#}:>{x,y}]&/@NestList[
Prepend[_], {y_,___},3])[list]]

(* {
{{x1,a1},{x2,a2},{x3,a3},{x4,a4},{x5,a5},{x6,a6},{x7,a7},{x8,a8},{x9,a9}},
{{x3,b3},{x4,b4},{x5,b5},{x7,b7},{x8,b8}},
{{x3,c3},{x4,c4},{x5,c5},{x8,c8}},
{{x5,d5}}
}
*)

{data1,data2,data3,data4}=={data1x,data2x,data3x,data4x}

(* True *)


list

(As given by @eldo):

list = {{x1, {a1}}, {x2, {a2}}, {x3, {a3, b3, c3}}, {x4, {a4, b4,
c4}}, {x5, {a5, b5, c5, d5, e5}}, {x6, {a6}}, {x7, {a7,
b7}}, {x8, {a8, b8, c8}}, {x9, {a9}}};

• Thanks. This was the approach I was thinking about too. Nice implementation. I couldn't figure out how to do the blanks programmaticly.
– Hugh
Commented Oct 5, 2023 at 14:17
 list = {{x1, {a1}}, {x2, {a2}}, {x3, {a3, b3, c3}}, {x4, {a4, b4,
c4}}, {x5, {a5, b5, c5, d5, e5}}, {x6, {a6}}, {x7, {a7,
b7}}, {x8, {a8, b8, c8}}, {x9, {a9}}};

list // MatrixForm


 data = <|Rule @@@ list|>


<|x1 -> {a1}, x2 -> {a2}, x3 -> {a3, b3, c3}, x4 -> {a4, b4, c4}, x5 -> {a5, b5, c5, d5, e5}, x6 -> {a6}, x7 -> {a7, b7}, x8 -> {a8, b8, c8}, x9 -> {a9}|>

get[n_] := KeyValueMap[List] @ DeleteMissing @ Query[All, n] @ data

Grid[{MatrixForm /@ Table[get[i], {i, 5}]}]


get[1]


{{x1, a1}, {x2, a2}, {x3, a3}, {x4, a4}, {x5, a5}, {x6, a6}, {x7, a7}, {x8, a8}, {x9, a9}}

get[2]


{{x3, b3}, {x4, b4}, {x5, b5}, {x7, b7}, {x8, b8}}

etc.

list =
{{x1, {a1}}, {x2, {a2}}, {x3, {a3, b3, c3}}, {x4, {a4, b4, c4}},
{x5, {a5, b5, c5, d5, e5}}, {x6, {a6}}, {x7, {a7, b7}},
{x8, {a8, b8, c8}}, {x9, {a9}}};


A variant of 1066's nice solution using FoldList and Query

With[{args = FoldList[Prepend[#1, Blank[]] &, {y_, ___}, Range[3]]},
Query[Cases[{x_, #} :> {x, y}] & /@ args] @ list]


{{{x1,a1},{x2,a2},{x3,a3},{x4,a4},{x5,a5},{x6,a6},{x7,a7},{x8,a8},{x9,a9}},
{{x3,b3},{x4,b4},{x5,b5},{x7,b7},{x8,b8}},
{{x3,c3},{x4,c4},{x5,c5},{x8,c8}},
{{x5,d5}}}

Using Comap (new in 14.0)

With[{args = FoldList[Prepend[#1, Blank[]] &, {y_, ___}, Range[3]]},
Comap[Cases[{x_, #} :> {x, y}] & /@ args, list]]


same result