3
$\begingroup$

Excuse me. This might be user specific question but I need your help. I have an indexed data, where positive integers are used for indexing data. I am trying to group data putting condition on index variable so that I can analyze group-wise data. But I don't know. Please guide me. Here is an example:

testdat = {{1, a11, b11}, {1, a12, b12}, {2, a21, b21}, {3, a31, 
   b31}, {4, a41, b41}, {2, a22, b22}, {3, a32, b32}, {3, a33, 
   b33}, {4, a42, b42}, {5, a51, b51}, {6, a61, b61}, {6, a62, 
   b62}, {5, a52, b52}, {5, a53, b53}, {5, b54, b54}};

If I do the following

GatherBy[testdat, First]

it groups data using first column as index. However, I want to group data in which rows indexed 1 and 2 should be one group, rows indexed 3,4 and 5 should be another group and row indexed 6 should be another group. Here is how the result should look like:

{{{1, a11, b11}, {1, a12, b12}, {2, a21, b21}, {2, a22, b22}}, {{3, 
   a31, b31}, {3, a32, b32}, {3, a33, b33}, {4, a41, b41}, {4, a42, 
   b42}, {5, a51, b51}, {5, a52, b52}, {5, a53, b53}, {5, b54, 
   b54}}, {{6, a61, b61}, {6, a62, b62}}}

Another way to group the data would be to put rows indexed 1,2 and 3 into one group and rows indexed 4,5 and 6 into another group like in the following result:

{{{1, a11, b11}, {1, a12, b12}, {2, a21, b21}, {2, a22, b22}, {3, a31,
    b31}, {3, a32, b32}, {3, a33, b33}}, {{4, a41, b41}, {4, a42, 
   b42}, {5, a51, b51}, {5, a52, b52}, {5, a53, b53}, {5, b54, 
   b54}, {6, a61, b61}, {6, a62, b62}}};

This is just the theory. I need code to perform this job. I expect to hearing form Great guys form this community. Thank you in advance.

$\endgroup$
2
$\begingroup$

We can MapIndexed over a list of group lists to create an indexing function g, then compose this with the test function fn:

gatherInto1[dat_, fn_, groups__List] :=
  Module[{g},
    MapIndexed[(g[#] = #2[[1]]) &, {groups}, {2}];
    GatherBy[dat, g @ fn @ # &]
  ]

Example:

gatherInto1[testdat, First, {1, 2}, {3, 4, 5}, {6}]
{
 {{1, a11, b11}, {1, a12, b12}, {2, a21, b21}, {2, a22, b22}},
 {{3, a31, b31}, {4, a41, b41}, {3, a32, b32}, {3, a33, b33}, {4, a42, b42},
   {5, a51, b51}, {5, a52, b52}, {5, a53, b53}, {5, b54, b54}},
 {{6, a61, b61}, {6, a62, b62}}
}

In Mathematica 10 we can use GroupBy and Lookup. (Fred Simons referenced this but did not provide a complete implementation.)

gatherInto2[dat_, fn_, groups__List] :=
  With[{asc = GroupBy[dat, fn]},
    Join @@ Lookup[asc, #, {}] & /@ {groups}
  ]

Each function has its place.

  • gatherInto1 works with patterns whereas gatherInto2 does not.
  • gatherInto2 can be quite a bit faster than gatherInto1.

Timings:

big = RandomInteger[{1, 6}, {500000, 3}];

gatherInto1[big, First, {1, 2}, {3, 4, 5}, {6}] // AbsoluteTiming // First
gatherInto2[big, First, {1, 2}, {3, 4, 5}, {6}] // AbsoluteTiming // First
0.524

0.029
$\endgroup$
  • $\begingroup$ As always, thank you so much, Mr. Wizard. $\endgroup$ – ramesh Feb 6 '15 at 14:37
2
$\begingroup$

One of the ways is the following:

GatherBy[testdat, Position[{{1, 2}, {3, 4, 5}, 6}, #[[1]]][[1]] &]

(* {{{1, a11, b11}, {1, a12, b12}}, {{2, a21, b21}, {2, a22, b22}}, {{3, a31, b31}, {3, a32, b32}, {3, a33, b33}}, {{4, a41, b41}, {4, a42, b42}}, {{5, a51, b51}, {5, a52, b52}, {5, a53, b53}, {5, b54, b54}}, {{6, a61, b61}, {6, a62, b62}}}  *)

For your second gathering, you can replace the first argument of Position with {{1,2,3}, {4,5,6}}.

It might be that the following is not exactly what you are looking for, but it shows something of the flexability of working with Associations. Here I construct the association:

asc=Association[ (First[#]->Rest[#])& /@ testdat]

(* <|1->{a12,b12},2->{a22,b22},3->{a33,b33},4->{a42,b42},5->{b54,b54},6->{a62,b62}|> *)

And here are the values with keys 1,2,3:

Lookup[asc,Key /@ {1,2,3}]

(*  {{a12,b12},{a22,b22},{a33,b33}} *)
$\endgroup$
  • $\begingroup$ @ Fred, thank you for your answer. What if I have a large data and index is too big to navigate. In fact, I have over 2m rows. $\endgroup$ – ramesh Dec 11 '14 at 15:22
  • $\begingroup$ @rka I have no experience at all with large data sets. But I understood that the function Lookup with Associations is very fast, and independent where you want to look in your data (beginning or end). I also understood that Associations are "memory hungry". My feeling is that if your data set fits in the memory of Mathematica as an Association, you will be better of with Lookup, of course depending of what you want to do. So I edited my answer showing how you could transform your testdat to an Association and how you could use LookUp. You might also look at the function GroupBy. $\endgroup$ – Fred Simons Dec 11 '14 at 15:59
  • $\begingroup$ @ Fred, thank you fro your suggestion. I will work on your function and suggestion. Thanks again. $\endgroup$ – ramesh Dec 11 '14 at 16:01
2
$\begingroup$
group = (Switch[#, 
                1 | 2 | 3, 1,
                _,         2
         ] &);


GatherBy[testdat, group[ #[[1]] ] & ]
{
   {{1, a11, b11}, {1, a12, b12}, {2, a21, b21}, {3, a31, b31}, {2, a22, b22},
    {3, a32, b32}, {3, a33, b33}}, 
   {{4, a41, b41}, {4, a42, b42}, {5, a51, b51}, {6, a61, b61}, {6, a62, b62}, 
    {5, a52, b52}, {5, a53, b53}, {5, b54, b54}}
}

and for oryginal request:

group = (Switch[#, 
                6,     1,
                1 | 2, 2, 
                _,     3] &);
{
    {{1, a11, b11}, {1, a12, b12}, {2, a21, b21}, {2, a22, b22}}, 
    {{3, a31, b31}, {4, a41, b41}, {3, a32, b32}, {3, a33, b33}, 
     {4, a42, b42}, {5, a51, b51}, {5, a52, b52}, {5, a53, b53}, {5, b54, b54}},
    {{6, a61, b61}, {6, a62, b62}}
 }
$\endgroup$
  • $\begingroup$ @ Kuba, thank you for your time to answer this question. $\endgroup$ – ramesh Dec 11 '14 at 15:55
  • $\begingroup$ @rka no problem, is it useful? :) $\endgroup$ – Kuba Dec 11 '14 at 16:15
  • $\begingroup$ @ I am trying to apply your code to my need but failing. Whether you could help me. I have data indexed form 40 to 1057. That is oldindex=Range[40,1057]. I have constructed newindex like newindex=Partition[oldindexindex, 3]; Now I want to group data according to new index. Any help $\endgroup$ – ramesh Dec 11 '14 at 16:54
  • 1
    $\begingroup$ In general, if you can find a function f that is constant on each of your groups of indices, then you can use that function in GatherBy with argument #[[1]]. In this special case GatherBy[dataset, Floor[(#[[1]] - 1)/3] &] will work. $\endgroup$ – Fred Simons Dec 11 '14 at 18:30
2
$\begingroup$
Cases[testdat, {Alternatives[##], __}] & @@@ {{1, 2}, {3, 4, 5}, {6}}

Update:

based on your comment you can do it using two methods: prepare your index before

index=Range[1,6,2]

and then

Cases[testdat, {Alternatives[##], __}] & @@@ {index}

you can also do it like this:

Flatten[Table[Cases[testdat, {i, __}], {i, 1, 6, 2}], 1]
$\endgroup$
  • $\begingroup$ @ Algohi, thank you for your answer. What if I have a large data and index is too big to navigate. In fact, I have over 2M obs. $\endgroup$ – ramesh Dec 11 '14 at 15:21
  • $\begingroup$ if you have too big data then cases is your choose because it is fast. i don't get what you mean be index is too big to navigate? as long as you have the criteria to group the rows then you can use this method. $\endgroup$ – Algohi Dec 11 '14 at 15:24
  • $\begingroup$ In your code, you have {{1, 2}, {3, 4, 5}, {6}} as criterion. Can we specify criteria as function than list? Like index is between two numbers: i<=index<=i+1, {i, 1, 6, 2}. Something like this. $\endgroup$ – ramesh Dec 11 '14 at 15:37
  • $\begingroup$ @ Algohi, thanks for your update. I think, this might serve my need. I am working on it thought on my real data set. $\endgroup$ – ramesh Dec 11 '14 at 16:06
  • $\begingroup$ you mean newindex=Partition[oldindex, 3]? use index instead of {index} in my code. like this: Cases[testdat, {Alternatives[##], __}] & @@@newindex $\endgroup$ – Algohi Dec 11 '14 at 16:38
1
$\begingroup$

Just for fun:

Join @@@ ({{1, 2}, {3, 4, 5}, {6}} /.Last@Reap[Sow[{##}, #1] & @@@ testdat, _, Rule])

yields:

{{{1, a11, b11}, {1, a12, b12}, {2, a21, b21}, {2, a22, b22}}, {{3, 
   a31, b31}, {3, a32, b32}, {3, a33, b33}, {4, a41, b41}, {4, a42, 
   b42}, {5, a51, b51}, {5, a52, b52}, {5, a53, b53}, {5, b54, 
   b54}}, {{6, a61, b61}, {6, a62, b62}}}
$\endgroup$
  • $\begingroup$ thanks for taking time to answer my question. $\endgroup$ – ramesh Feb 6 '15 at 14:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.