# Timing and memory use is critical:fast partitioning of binary sparse array

I do experiments were I measure a signal, and I need to know at what time points it is above a threshold. If at a given time point, the signal is above the threshold, then a '1' gets put into the array otherwise the value is zero. This effectively generates a binary signal.

A SparseArray is a natural solution to this because I want to conserve memory because the data can be quite large. For example, 1000 raw signals, each with 60,000 to 100,000 time points.

After putting the data into a SparseArray, I want split the array into groups of 1's and get the positions where the Array was split.

In a simple example I make a sparse array that represents 20 seconds of time

 lowArray =
SparseArray[# -> 1 & /@ {1, 2, 3, 4, 5, 10, 11, 15, 16, 17, 18, 19, 20}, 20]

ListPlot[lowArray, Joined -> True] I would like the output to be two lists. One which contains the splits and one that contains the positions

 splits = { {1,1,1,1,1}, {1,1}, {1,1,1,1,1};
pos = {{1,5},{10,11}, {15,20}}


This question is similar to Find Continuous Sequences Inside a List, but Split does not work on a SparseArray. Also, its not clear to me how to get the positions where the array was split.

Solutions do not have to use a SparseArray, but it is my preference because the data can be quite large. However if this can be done faster not using a SparseArray', the faster method will be preferred.

## 1 Answer

pos={First@#,Last@#}&/@Split[Flatten@lowArray["NonzeroPositions"],#2-#1==1&]


{{1,5},{10,11},{15,20}}

splits=lowArray[[Range@@#]]&/@pos//Normal


{{1,1,1,1,1},{1,1},{1,1,1,1,1,1}}`