2
$\begingroup$

I have a large collection of data with multi levels. An example of the data (in Association shape) is here.

The data looks like this:

data = ToExpression[Import["test"]];
data[[;; , ;; 3]]

enter image description here

I am looking for fast way to extract the data from all Keys when the time (the last column) is within certain values.

If you see the following plot, I am looking to extract the data from all Keys within 5 seconds around each peaks when the peaks are above certain values compare to certain Key (say Key "G32" when the values are more than 20).

    DateListPlot[data[[;; , ;; 200, {-1, 2}]], PlotRange -> All, 
 GridLines -> Automatic]

enter image description here

If I choose "G32" as my reference for selection then the time that I need to extract the data around is:

    allevents = 
  First /@ Split[
    Select[data[["G32"]], #[[2]] > 20 &], #2[[-1]] - #1[[-1]] < 5 &];

and this result is like this:

DateListPlot[data[[;; , ;; 900, {-1, 2}]], PlotRange -> All, 
 GridLines -> Automatic, 
 Epilog -> {PointSize[0.01], Point[allevents[[;; , {-1, 2}]]]}]

enter image description here

What I am facing is the difficulty to efficiently and quickly extract the data around each time of allevents within 5 seconds. Currently I am using this method but it takes too long:

alleventstime = allevents[[;; , -1]];
AbsoluteTiming[
 eventsdata = 
   Table[Select[#, Abs[#[[-1]] - i] <= 5 &], {i, alleventstime}] & /@ 
    data;]
{252.585, Null}

Part of the result looks like this: enter image description here

Any idea how can I do that fast. I have tried couple of other methods but no much luck.

Thank you

$\endgroup$
  • $\begingroup$ @Edmund, the data is provided in the link at the beginning of the question $\endgroup$ – Algohi Aug 11 '17 at 3:08
2
$\begingroup$

You may use PeakDetect to find the peaks combined with the Date & Time and Interval Arithmetic guides to reduce the evaluation time.

The time is being extended because you are making a complete pass of each set of data for each peak window. A method that makes only one pass for all peak windows will significantly drop the time.

With data as defined in the OP.

Then the location of the peaks can be obtained by

peakLocs = Position[1]@PeakDetect[data[["G32", All, 2]], Automatic, Automatic, 20]
{{45},{56},{98},{166},{190},{195},{342},{352},{396},{465},{514},
 <<369>>,
 {17571},{17658},{17663},{17674},{17704},{17729},{17732},{17773},{17786},{17811},{17823}}

DatePlus can be used with Interval to create a 5 second interval about these peaks. IntervalUnion can then join these Intervals into one.

intervalWindows =
  Interval@
      {
       AbsoluteTime@DatePlus[DateObject@data[["G32", #, -1]], {-5, "Second"}],
       AbsoluteTime@DatePlus[DateObject@data[["G32", #, -1]], {5, "Second"}]
      } & /@
     Flatten@peakLocs // Apply[IntervalUnion];
Interval[{3.69061*10^9,3.69061*10^9},{3.69061*10^9,3.69061*10^9},
    <<372>>,
    {3.70582*10^9,3.70582*10^9}]

IntervalMemberQ can be used with Select to make one pass on each set of data that returns all items in the peak windows. This runs in less than a second.

selected = Select[IntervalMemberQ[intervalWindows, Last@#] &] /@ data;

A quick visual check indicates that all is well.

DateListPlot[selected[[All, ;; 50, {-1, 2}]], Joined -> False]

Mathematica graphics

Hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.