Partitioning time series data in sublists by their duration

I would like to do calculations on irregularly sampled time series data at intervals - for example on an hourly basis - rather than over the whole time series.

The entire dataset is split up by time into sublists of continuous data, so there are no large gaps in time within a sublist. Each data point includes a time value, but the data are not uniformly distributed in time, so the index of the numbers cannot be used as a proxy for duration and I could not find a way to use Partition.

The data is in a list of several sublists (varying length) - these are the continuous datasets. I'd like to keep these sublists, but do the calculations at intervals (e.g. hourly) within the sublists. Each continuous dataset contains another set of sublists, one for each data point, in which there are elements (a fixed number) corresponding to each measurement type and the time.

I think I need to somehow split the data first by duration into another layer of sublists, and then apply the same calculations to the sublists.

For more information: The code I currently have assigns names to each data type and calculates background values within the second layer of lists as follows.

co=#[[All,1]]&/@mainlist;
co2=#[[All,3]]&/@mainlist;

co2int=#[0]&/@(LinearModelFit[{#[[All,1]],#[[All,3]]}tr,{1,x},x]&/@mainlist;
(*calculating the intercept using two measurement types*)
co2corr=co2-co2int;
(*using the intercept to correct CO2 background*)

times=#[[All,15]]&/@mainlist;


I'd like to do similar calculations but for hourly datasets within each sublist rather than to each sublist as a whole.

The data structure looks something like this:

mainlist=
{
{(*first cts dataset*)
(*{co,coerr,co2,co2err,...,{yyyy,MM,dd,hh,mm,ss}}*)
{2*10^18,0.3,2*10^20,0.41,...{2004,12,18,04,18,22}},
...,
{3*10^18,0.35,2.1*10^20,0.42,...{2004,12,18,06,19,42}}
},
...(*further cts datasets*)
{(*final cts dataset*)
{2.3*10^18,...,{2004,12,29,19,4,53}},
...
}
}


Many thanks.

-
What is the format of the times? HH24:MI:SS? As an aside why are you using this construct co=#[[All,1]]&/@mainlist - full marks for functional form but co=mainlist[[All,1]] is a bit more approachable. –  Ymareth Feb 3 '14 at 12:43
The time format is {YYYY,MM,DD,hh,mm,ss} but could be converted to absolute time. The functional form is so that I can access each of the sublists (i.e. each piece of continuous time series data) within the main list of data - if I use co=mainlist[[All,1]] I lose a layer of sublists. I might be able to get around this by using three indices, but functional form seemed more straightforward. Thanks! –  VFT Feb 3 '14 at 13:08
My mistake, I thought you'd written Apply not Map (there's probably a built-in function for shame). So by the time you're at co2int=#[0]&/@(LinearModelFit[{#[[All,1]],#[[All,3]]}tr,{1,x},x]&/@mainlist; each of the #[[All,?]] are simple lists of numbers? –  Ymareth Feb 3 '14 at 13:47
There are still sublists (again to separate continuous datasets) but the #[[All,?]] separates out the data types. So co is in the form {{ctsdataset1},{ctsdataset2},...} where each ctsdataset is itself a list of CO measurements. –  VFT Feb 3 '14 at 13:54
Can you supply a simple example of the hierarchy of your data structure? It would help sleepy guys like me think more easily about the problem. Even better, if you can give us a simple function to generate some random data we could use to help design a solution. That said, my first instinct on this goes to creating a dispatch table. See Dispatch[] and @whuber's answer link. A dispatch table just might give you the versatility you need to do what you want to do. –  Jagra Feb 3 '14 at 14:31

1 Answer

An extended comment / partial answer. I'd suggest pulling this apart a bit.

First a function to perform the fit.

fit[data_List]:=LinearModelFit[{data[[All,1]],data[[All,3]]}tr,{1,x},x];


So your original main piece would now read (btw you seem to have a missing ")" in there.)...

co2int=#[0]&/@fit[#]&/@mainlist;


Now you can write another function to map fit over a list split by hour...

fitSplit[data_list]:=Map[fit,SplitBy[data, #[[15,2]]&];


Slotting this in instead of fit.

-
Thanks! This seems like it should work, but I still run into problems when trying to implement it - one layer, the one in which the data are split by hour, still gets lost. So co2int stays the same if I remove SplitBy[data,#[[15,2]]&; in the definition of fitSplit, and just replace it with dat. This could be a problem with my use of SplitBy, but I think the real problem is my clumsy data structure, so perhaps restructuring into a dispatch table could simplify things. –  VFT Feb 15 '14 at 3:05
My bad, hours are the 4th not 2nd index. Try fitSplit[data_list]:=Map[#[[1,15,4]]->fit[#]&,SplitBy[data, #[[15,4]]&]; Which will split correctly and also label each fit by the hour its from. –  Ymareth Feb 15 '14 at 10:49
Perfect - based on this, I could get exactly what I was looking for. I incorporated calculation and subtraction of the background intercept into the definition of fit, to get: fit[dat_List]:=#[[All,3]]&@dat-(#[0]&@ (LinearModelFit[{dat[[All,1]],dat[[All,3]]}tr,{1,x},x]); co2corr=fitSplit[#]&/@mainlist; Flatten[co2corr,{{1},{2,3}}] –  VFT Feb 17 '14 at 13:50