1
$\begingroup$

I have 2 data sources with different data acquisition rates. My first data source is a temperature measurement over time with a resolution of 1 s. My second source is a spectrometer with 20 ms time resolution. I would like to synchronize the output of these two different instruments.

My initial approach was to use FromDateString to convert everything into a DateTime object, and perform comparisons using Nearest. This was very slow with the amount of data that I have (~11k data points in the 1st source, and ~5k data points in the 2nd data source).

I have a solution that "works" now, but it's less than ideal. I was hoping to use UnixTime, but that only has seconds resolution instead of milliseconds.

Ideally I would be able to do something like this:

temperatureData = Import[SystemDialogInput["FileOpen"]];
spectrometerData = Import[SystemDialogInput["FileOpen"]];
temperatureTimes = Map[FromDateString,temperatureData[[;;,1]]];
spectrometerTimes = Map[FromDateString,spectrometerData[[;;,1]]];
startIndex = Position[temperatureTimes,Nearest[temperatureTimes,spectrometerTimes[[1]]][[1]]][[1]];
finalIndex = Position[temperatureTimes,Nearest[temperatureTimes,spectrometerTimes[[-1]]][[1]]][[1]];
interpTemperature = Interpolation[
  {
    temperatureTimes[[startIndex;;finalIndex]],
    temperatureData[[startIndex;;finalIndex,2]]
  }//Transpose
];
(* This should yield a function which takes a DateTime object as the argument and returns an interpolated temperature value, so the following should be possible *)
interpTemperature[spectrometerTimes[[1]]];

This approach doesn't work, but I'm wondering if there's a proper approach. There doesn't seem to be another format like UnixTime but with ms resolution. Does this need to be a custom implementation where I define something like "ms since t0 of spectrometerData"?

Also, any guidance on cleaning up those Position calls would be appreciated. I know there's got to be a better way, I just haven't found it yet.

EDIT: I performed the interpolation using a custom time identifier and it seems to work OK

msSincet0Spectrometer = Table[
  QuantityMagnitude[
    UnitConvert[
      spectrometerTimes[[i]]-spectrometerTimes[[1]],
      "Milliseconds"
    ]
  ]
];
msSincet0Temperature = Table[
  QuantityMagnitude[
    UnitConvert[
      temperatureTimes[[i]]-spectrometerTimes[[1]],
      "Milliseconds"
    ]
  ]
];
interpTemperature = Interpolation[
  {msSincet0Temperature,temperatureData[[startingTemperatureIndex+1;;finalTemperatureIndex+1,2]]}//Transpose
];

This works as I would expect, but loses the absolute timing.

EDIT 2: My previous example was poor, so I'm going to provide a better one, and demonstrate that the answer in @lericr's comment works.

For the temperature data and spectrometer we can make fake data like so:

fakeTemperatureData = Table[
  {FromDateString["6/22/2023 14:27:12.1500"]+Quantity[i,"Seconds"],Sin[i/Pi]},
  {i,1,100}
];
fakeSpectrometerData = Table[
  {FromDateString["6/22/2023 14:28:12.5000"]+Quantity[i*20,"Milliseconds"],Sin[i/Pi]+Cos[i/Pi]},
  {i,1,500}
];
DateListPlot[{fakeTemperatureData,fakeSpectrometerData},Joined->False]

This generates two datasets with different acquisition rates, where one is completely contained within the other. To accomplish what I want, the slower acquisition must be converted into TimeSeries and then Interpolation can be applied:

interpTemperature = Interpolation[TimeSeries[fakeTemperatureData]];

Then, it's possible to query data points that correspond to the more rapid spectrometer data using AbsoluteTime (per @lericr):

interpTemperature[AbsoluteTime[fakeSpectrometerData[[1,1]]]];
(* output should be ~ 0.354113 *)
$\endgroup$
8
  • 2
    $\begingroup$ You can provide a granularity argument to DateObject, and one possibly granularity is "Instant". I assume you could use that to normalize across your datetime data. As for the interpolation bit, I think I'm grokking what you're trying to do, but it would be easier if you provided actual sample input data and expected results. As it stands now, you're expecting us to reverse engineer our own test data by interpreting your code. $\endgroup$
    – lericr
    Jun 23, 2023 at 15:55
  • $\begingroup$ I can add some test data. The full data is large, but I can add some test data. I've also figured out a hack using msSincet0 and I can provide some expected results. $\endgroup$ Jun 23, 2023 at 16:22
  • 1
    $\begingroup$ I think you might benefit by looking into TimeSeries. $\endgroup$
    – lericr
    Jun 23, 2023 at 16:29
  • 1
    $\begingroup$ If I'm understanding correctly, all you really need is an interpolation function for your temperature data. Using TimeSeries and Interpolation should get you that, I think. I think the whole normalizing to milliseconds is a red-herring. But maybe I'm missing something that will be clarified with test cases. $\endgroup$
    – lericr
    Jun 23, 2023 at 16:31
  • 2
    $\begingroup$ Yeah, that's confusing. Wrap your date argument with AbsoluteTime. Although the units are seconds, I believe AbsoluteTime will give fractions of seconds. $\endgroup$
    – lericr
    Jun 23, 2023 at 16:39

1 Answer 1

3
$\begingroup$

If I've understood the question and comments, you could proceed as follows. Let's assume you have myTempData as a list of time-value pairs, where each time element is an actual DateObject expression. You can then get an interpolation function like this:

myTempDataInterpolation = Interpolation[TimeSeries[myTempData]]

To use it, you'll need to wrap inputs with AbsoluteTime:

myTempDataInterpolation[AbsoluteTime[Now]]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.