3
$\begingroup$

This query segments a time-series and filters out dropout frames.

Can it be simplified - ie, made more readable?

segmentationQuery = 
 Query[SplitBy[#, NumberQ[#"POR_X"] &] &  ] /* 
  Query[Merge[Join] /* Key[True], GroupBy[NumberQ[#"POR_X"] &]]

(Note there's no SplitBy operator form in 10.0.0. hence using functional form.)

The Slot "POR_X" is used twice: in SplitBy and GroupBy - this is not pleasing.

Using only GroupBy in the top-most subquery yields only 2 groups of data (signal and blank) but all the numeric frames are merged (not the intended behavior).

This fragment, extracted from eye-tracking dataset, illustrates droputs of variable duration and number of rows:

data = {<|"Time" -> 0.033452, "POR_X" -> 613.76, 
  "POR_Y" -> 424.74|>, <|"Time" -> 0.066747, "POR_X" -> 611.84, 
  "POR_Y" -> 424.84|>, <|"Time" -> 0.100124, "POR_X" -> 610.53, 
  "POR_Y" -> 420.11|>, <|"Time" -> 0.133515, "POR_X" -> 605.95, 
  "POR_Y" -> 421.18|>, <|"Time" -> 2.01096, "POR_X" -> "", 
  "POR_Y" -> ""|>, <|"Time" -> 2.04425, "POR_X" -> "", 
  "POR_Y" -> ""|>, <|"Time" -> 2.07772, "POR_X" -> "", 
  "POR_Y" -> ""|>, <|"Time" -> 2.24451, "POR_X" -> 412.47, 
  "POR_Y" -> 805.92|>, <|"Time" -> 2.27792, "POR_X" -> 518.11, 
  "POR_Y" -> 652.38|>} //Dataset;

enter image description here

data[segmentationQuery][All, Dataset] // Normal

enter image description here

$\endgroup$

2 Answers 2

4
$\begingroup$

The following expression is slightly shorter:

data[
   SplitBy[#, NumberQ[#"POR_X"]&] &
][ Select[#[[1, "POR_X"]] =!= "" &], Dataset
] // Normal

dataset screenshot

Or, expressed as a Query:

query2 =
  Query[SplitBy[#, NumberQ[#"POR_X"]&] &] /*
  Query[Select[#[[1, "POR_X"]] =!= "" &], Dataset];
$\endgroup$
1
$\begingroup$

Small advice: use Missing[] instead of whitespace character. And dataset look better enter image description here

PS: if you have deal with time series probably better use TemporalData or TimeSeries

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.