# How can AbsoluteTimes be split by date faster?

I have data where the date is stored as an AbsoluteTime. Here's a dummy dataset:

sampleData=Transpose[{#,Range[Length[#]]}]&[Range[AbsoluteTime[{2012,10,01}],
AbsoluteTime[{2013,09,30}],60]];


I want to split it by date, so I use:

splitData=SplitBy[sampleData,DateList[First[#]][[;;3]]&];


This takes ~22 s on my machine. Often I have bigger data sets than this. Is there a faster way to get splitData?

-
Ooh... DateList is a killer. I once wrote something similar to what you need. Let me try to dig it up. –  The Toad Sep 30 '13 at 18:51

When you have your date/time in AbsoluteTime and need to compare two dates repeatedly/several times, it is faster to count increments of seconds/minutes/hours than use DateList in each.

For simplicitly, I'll assume that the list is sorted by date and that the first element corresponds to the earliest date. We'll take our reference time as 00:00 hrs on this date. Then, splitting the data by date is as simple as:

With[{start = AbsoluteTime@DateList[sampleData[[1, 1]]][[;; 3]]},
splitData2 = GatherBy[sampleData, Quotient[#[[1]] - start, 86400] &]
]; // AbsoluteTiming
(* 0.055087, Null *)


Comparing this with your DateList solution (note that I've corrected the test function in SplitBy to actually do what you described):

splitData = SplitBy[sampleData,DateList[First@#][[;;3]]&]; // AbsoluteTiming
(* {19.206535, Null} *)

splitData == splitData2
(* True *)

-
+1. But it might not be as simple as that, in general - there are leap seconds and the like. –  Leonid Shifrin Sep 30 '13 at 19:06
@LeonidShifrin Absolutely. Leap seconds are not accounted for in this (and other date-time oddities). I suppose if OP's data has something like that, then a JLink/Java approach similar to what you've answered before might be a better solution. –  The Toad Sep 30 '13 at 19:08
I might add some answer like that later. I happened to have recently used Java / reloader again for a similar practical problem ( I was helping friends to parse a 4Gb log file from their server, using Mathematica, and going to Java for date parsing was a crucial point - everything else I was able to do reasonably well within Mathematica). And by the way, my recent work has been on Mac, so I finally fixed the reloader for Mac too (at least it works on my system now, while used to produce the same error that you have reported before). –  Leonid Shifrin Sep 30 '13 at 19:24
Oh, that's good! Is the updated version in your answer here? BTW, a lot of your tools might be better on github, which can host the latest version... I know you like gists, but it's not easy to search through them :) –  The Toad Sep 30 '13 at 19:32
Re: updated version - not yet, but will be soon, I just need to repost it. Re: GitHub - yes, I do have a few repos there, and I agree that it is convenient. If only I could finish my stuff, it would have search built in the system, on Mathematica-side. It has been very close to completion, for several months now. The time is very scarce resource these days for me, though - you could have noticed it by how much I reduced my activity here in the last several months. But I still hope to get that stuff out soon. –  Leonid Shifrin Sep 30 '13 at 20:26

Here is another option using Floor:

(splitData=SplitBy[sampleData,DateList[First[#]][[;;3]]&])//AbsoluteTiming//First
(splitData2=SplitBy[sampleData,Floor[#[[1]]/86400.]&])//AbsoluteTiming//First
(splitData3=GatherBy[sampleData,Floor[#[[1]]/86400.]&])//AbsoluteTiming//First


9.618810

2.227605

0.056786

splitData == splitData2 == splitData3


True

PS: use 86400. instead of 86400 make all the difference for splitData3

-