Partitioning time series data in sublists by their duration

Question

I would like to do calculations on irregularly sampled time series data at intervals - for example on an hourly basis - rather than over the whole time series.

The entire dataset is split up by time into sublists of continuous data, so there are no large gaps in time within a sublist. Each data point includes a time value, but the data are not uniformly distributed in time, so the index of the numbers cannot be used as a proxy for duration and I could not find a way to use Partition.

The data is in a list of several sublists (varying length) - these are the continuous datasets. I'd like to keep these sublists, but do the calculations at intervals (e.g. hourly) within the sublists. Each continuous dataset contains another set of sublists, one for each data point, in which there are elements (a fixed number) corresponding to each measurement type and the time.

I think I need to somehow split the data first by duration into another layer of sublists, and then apply the same calculations to the sublists.

For more information: The code I currently have assigns names to each data type and calculates background values within the second layer of lists as follows.

co=#[[All,1]]&/@mainlist;
co2=#[[All,3]]&/@mainlist;

co2int=#[0]&/@(LinearModelFit[{#[[All,1]],#[[All,3]]}tr,{1,x},x]&/@mainlist;
(*calculating the intercept using two measurement types*)
co2corr=co2-co2int;
(*using the intercept to correct CO2 background*)

times=#[[All,15]]&/@mainlist;

I'd like to do similar calculations but for hourly datasets within each sublist rather than to each sublist as a whole.

The data structure looks something like this:

mainlist=
{
  {(*first cts dataset*)
    (*{co,coerr,co2,co2err,...,{yyyy,MM,dd,hh,mm,ss}}*) 
    {2*10^18,0.3,2*10^20,0.41,...{2004,12,18,04,18,22}},
    ...,
    {3*10^18,0.35,2.1*10^20,0.42,...{2004,12,18,06,19,42}}
  },
  ...(*further cts datasets*)
  {(*final cts dataset*) 
    {2.3*10^18,...,{2004,12,29,19,4,53}},
    ...
  }
}

Many thanks.

Answer 1

An extended comment / partial answer. I'd suggest pulling this apart a bit.

First a function to perform the fit.

fit[data_List]:=LinearModelFit[{data[[All,1]],data[[All,3]]}tr,{1,x},x];

So your original main piece would now read (btw you seem to have a missing ")" in there.)...

co2int=#[0]&/@fit[#]&/@mainlist;

Now you can write another function to map fit over a list split by hour...

fitSplit[data_list]:=Map[fit,SplitBy[data, #[[15,2]]&];

Slotting this in instead of fit.