7
$\begingroup$

I've been trying to separate large segments of DateList formatted data (years worth at a time) into individual days using the Split function.
The data has the form: 

data = {{{y,m,d,h,m,s}, 154},{{y,m,d,h,m,s}, 157},...}

and unfortunately there is not a uniform length of data per day (or I would have just partitioned it). This is what I've been trying (or variations of it):

splitData = Split[data,#2[[1,3]]>#1[[1,3]]&];

or

splitData = Split[data,Less[#1[[1,3]],#2[[1,3]]]];

and plenty of other versions, but it doesn't split where I need it to.

I thought if I compared the days in the DateList slot the data would split (as days of a month are increasing until the new month begins.

My current solution (which is very inefficient) pulls the dates out of data and applies Union, Union@data[[All,1]], then iterates through each data point comparing dates using nested tables. ... I know, a very poor method.

Anyone see how to us Split or another more efficient method to separate?

If you need test data to work with I've been using this instead of the real data (a much smaller set):

dpmo[month_, year_] := DateDifference[{year, month, 1}, {If[month == 12, year + 1, year], If[month == 12, 1, month + 1], 1}];
dates = Flatten[Table[Table[{2012, month, day, 0, 0, 0}, {day, 1, dpmo[month, 2012]}], {month, 1, 12}], 1];
data = Flatten[Table[Table[{dates[[m]], RandomInteger[{100, 250}]}, {100}], {m, 1, Length@dates}], 1];

The code generates the equivalent to a years worth of data, however there is a definitive set of data per day (100) which I do not have in the actual set. I didn't feel like adding that to the test data.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
8
$\begingroup$

You can use Gather which will do all the work for you:

Gather[data, #1[[1, 1 ;; 3]] == #2[[1, 1 ;; 3]] &]

{{{{2012, 1, 1, 0, 0, 0}, 1}, {{2012, 1, 1, 0, 0, 1}, 2}, {{2012, 1, 1, 0, 1, 0}, 3}}, {{{2012, 1, 2, 0, 0, 0}, 4}, {{2012, 1, 2, 0, 0, 1}, 5}}, {{{2012, 1, 3, 0, 0, 0}, 6}}}

For an alternative way of generating a range of dates, should you need them, you could try this approach with a start date and a number of days required:

DatePlus[{2012, 12, 25, 0, 0, 0}, #] & /@ Range[0,9]

{{2012, 12, 25, 0, 0, 0}, {2012, 12, 26, 0, 0, 0}, {2012, 12, 27, 0, 0, 0}, {2012, 12, 28, 0, 0, 0}, {2012, 12, 29, 0, 0, 0}, {2012, 12, 30, 0, 0, 0}, {2012, 12, 31, 0, 0, 0}, {2013, 1, 1, 0, 0, 0}, {2013, 1, 2, 0, 0, 0}, {2013, 1, 3, 0, 0, 0}}

Improve this answer
$\endgroup$
6
  • $\begingroup$ Definitely better than mine, if the data is not ordered and we want the data ordered. $\endgroup$
    – Mark McClure
    Commented Sep 18, 2012 at 15:58
  • $\begingroup$ I'm not sure which is best, if for some reason the dates aren't ordered then won't SplitBy fail to group values for the same day which aren't successive, just as Gather will only order the values according to their position in the data? Perhaps both need a pre(post)-sorting phase. Quick timing check seemed to suggest Gather had the edge on speed. $\endgroup$
    – image_doctor
    Commented Sep 18, 2012 at 16:32
  • 1
    $\begingroup$ Yes, SplitBy would definitely not group things by date, if the dates were out of order. Depending on the application, we might or might not want this behavior. For example, we might want different groups colored differently. $\endgroup$
    – Mark McClure
    Commented Sep 18, 2012 at 16:37
  • $\begingroup$ I didn't check timing. My guess would be that SplitBy would be faster since it does no re-ordering. Could be wrong, though. $\endgroup$
    – Mark McClure
    Commented Sep 18, 2012 at 16:40
  • $\begingroup$ Thanks for the responses, Gather seems to be doing the job perfectly, and although it's a lot of data the system seems to be doing it with little trouble and since it only has to parse it once (instead of continually) I'm satisfied. $\endgroup$
    – MRN16
    Commented Sep 18, 2012 at 17:09
7
$\begingroup$

I believe that SplitBy works. Using very simple data:

data = {
 {{2012,1,1,0,0,0},1},{{2012,1,1,0,0,1},2},{{2012,1,1,0,1,0},3},
 {{2012,1,2,0,0,0},4},{{2012,1,2,0,0,1},5},
 {{2012,1,3,0,0,0},6}
};
divided = SplitBy[data,#[[1,1;;3]]&];
Column[divided,Dividers -> All]

enter image description here

Note that SplitBy[data,f] splits between x1 and x2 when f[x1]=!=f[x2]. In the case here, I've taken #[[1,1;;3]]& which corresponds to the day portion of the date with data format.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.