Efficient and fast way to extract information from multiple levels data

Question

I have a large collection of data with multi levels. An example of the data (in Association shape) is here.

The data looks like this:

data = ToExpression[Import["test"]];
data[[;; , ;; 3]]

I am looking for fast way to extract the data from all Keys when the time (the last column) is within certain values.

If you see the following plot, I am looking to extract the data from all Keys within 5 seconds around each peaks when the peaks are above certain values compare to certain Key (say Key "G32" when the values are more than 20).

    DateListPlot[data[[;; , ;; 200, {-1, 2}]], PlotRange -> All, 
 GridLines -> Automatic]

If I choose "G32" as my reference for selection then the time that I need to extract the data around is:

    allevents = 
  First /@ Split[
    Select[data[["G32"]], #[[2]] > 20 &], #2[[-1]] - #1[[-1]] < 5 &];

and this result is like this:

DateListPlot[data[[;; , ;; 900, {-1, 2}]], PlotRange -> All, 
 GridLines -> Automatic, 
 Epilog -> {PointSize[0.01], Point[allevents[[;; , {-1, 2}]]]}]

What I am facing is the difficulty to efficiently and quickly extract the data around each time of allevents within 5 seconds. Currently I am using this method but it takes too long:

alleventstime = allevents[[;; , -1]];
AbsoluteTiming[
 eventsdata = 
   Table[Select[#, Abs[#[[-1]] - i] <= 5 &], {i, alleventstime}] & /@ 
    data;]
{252.585, Null}

Part of the result looks like this:

Any idea how can I do that fast. I have tried couple of other methods but no much luck.

Thank you

Answer 1

You may use PeakDetect to find the peaks combined with the Date & Time and Interval Arithmetic guides to reduce the evaluation time.

The time is being extended because you are making a complete pass of each set of data for each peak window. A method that makes only one pass for all peak windows will significantly drop the time.

With data as defined in the OP.

Then the location of the peaks can be obtained by

peakLocs = Position[1]@PeakDetect[data[["G32", All, 2]], Automatic, Automatic, 20]

{{45},{56},{98},{166},{190},{195},{342},{352},{396},{465},{514},
 <<369>>,
 {17571},{17658},{17663},{17674},{17704},{17729},{17732},{17773},{17786},{17811},{17823}}

DatePlus can be used with Interval to create a 5 second interval about these peaks. IntervalUnion can then join these Intervals into one.

intervalWindows =
  Interval@
      {
       AbsoluteTime@DatePlus[DateObject@data[["G32", #, -1]], {-5, "Second"}],
       AbsoluteTime@DatePlus[DateObject@data[["G32", #, -1]], {5, "Second"}]
      } & /@
     Flatten@peakLocs // Apply[IntervalUnion];

Interval[{3.69061*10^9,3.69061*10^9},{3.69061*10^9,3.69061*10^9},
    <<372>>,
    {3.70582*10^9,3.70582*10^9}]

IntervalMemberQ can be used with Select to make one pass on each set of data that returns all items in the peak windows. This runs in less than a second.

selected = Select[IntervalMemberQ[intervalWindows, Last@#] &] /@ data;

A quick visual check indicates that all is well.

DateListPlot[selected[[All, ;; 50, {-1, 2}]], Joined -> False]

Hope this helps.