# Replacing missing absolute time with NA

I have a data set absolute time data {absolute time, species} set that looks like this

mydata = {{3558297660,74}, {3558297720,38}, {3558297780,61},
{3558297900,26}, {3558297960,100}, {3558298020,50}, {3558298080,70},
{3558298140,90}, {3558298260, 29}, {3558298320,65}, {3558298380,24},
{3558298500,67},{3558298560,80}, {3558298740,16}, {3558298800,68}};


Each time step is 60 s apart, I'm trying to create a continuous data array spaced 60 s apart with missing time entries replaced by the right time and "NA" as below

newdata = {{3558297660,74}, {3558297720,38}, {3558297780,61},
{3558297840,"NA"}, {3558297900,26}, {3558297960,100}, {3558298020,50},
{3558298080,70}, {3558298140,90}, {3558298200,"NA"}, {3558298260,29},
{3558298320,65}, {3558298380,24}, {3558298440,"NA"}, {3558298500,67},
{3558298560,80}, {3558298620,"NA"}, {3558298680,"NA"}, {3558298740,16},
{3558298800,68}};


The main reason why I have sought help of the community is that I intend to use the DateListPlot function to plot a TemporalData using the Joined option which is annoying when there are gaps in the data (as it joins this point, making the plot a bit messy). I have adapted your code for what I want and was wondering if there are options in DateListPlot to handle this. I will to plot the data using the joined option but skipping any large time gap or as replacing them with "NA"/Missing["NA"]

• I would suggest using Missing["NA"] instead, as that's the format easily handled by Mathematica. – J. M. is away Aug 22 '17 at 16:23

An alternative to Alexey Golyshev's answer that is more functional programming oriented:

data = <|Rule @@@ mydata|>;
{#, Lookup[data, #, Missing["NA"]]} & /@
Range[Sequence @@ MinMax[mydata[[All, 1]]], 60]


The idea is to create a lookup table from your data and to create a new list using which is filled using Lookup.

• Cool! Short and elegant code. +1 – Alexey Golyshev Aug 22 '17 at 18:01
• Thanks @Mathe172. Really helpful – L Pola Aug 24 '17 at 10:51
ts = TimeSeriesResample[mydata, 60, ResamplingMethod -> {"Constant", "NA"} ];

ts == newdata


True

DateListPlot[ts]


newdata = Reap[
Sow[mydata[[1]]];
Do[
diff = mydata[[i + 1, 1]] - mydata[[i, 1]];
If[diff > 60,
Sow /@ Table[{mydata[[i, 1]] + 60*j, Missing["NA"]}, {j, Quotient[diff, 60] - 1}]
];
Sow[mydata[[i + 1]]]
,
{i, 1, Length@mydata - 1}
]
][[2, 1]]

DateListPlot[
TimeSeries[newdata, MissingDataMethod -> {"Interpolation", InterpolationOrder -> 1}]
]


• Thanks @Alexey, managed to get it to work. The main reason why I have sought help of the community is that I intend to use the DateListPlot function to plot a temporal data using the Joined option which is annoying when there are gaps in the data (as it joins this point, making the plot a bit messy). I have adapted your code for what I want and was wondering if there are options in DateListPlot to handle this. – L Pola Aug 24 '17 at 10:51
• @LPola See updated answer. – Alexey Golyshev Aug 24 '17 at 14:20
• I got the previous code to work for what I intended. I want to join the continuous point in my data and leave the gaps without interpolating. My amended question was just to check if there's a simpler way to do this using the DateListPlot. But thanks once again for helping. – L Pola Aug 25 '17 at 9:34

This a solution using the option ResamplingMethod for TimeSeries along with the TimeSeriesResample built-in:

Block[{ts, tsopts, ts2, correctQ},
With[{tspec = {"Year", "/", "Month", "/", "Day", ", ", "Hour", ":", "Minute", ":", "Second", ".", "Millisecond"}},
tsopts = Sequence[
ResamplingMethod -> Missing["NotAvailable"],
DateFunction -> (DateString[#, tspec] &)
];

ts = Apply[TimeSeries[#2, {#1}, Evaluate[tsopts]] &][Transpose[mydata]];
correctQ = ts2["Path"] == (newdata /. "NA" -> Missing["NotAvailable"]);

DateListPlot[
{ts, ts2},
PlotLabel -> suggestion == HoldForm[newdata] -> correctQ,
PlotLegends -> {HoldForm[mydata], suggestion}
]
]
]


Alternatively, modifying tsopts above like in

tsopts = Sequence[
ResamplingMethod -> {"Interpolation", InterpolationOrder -> 0},
DateFunction -> (DateString[#, tspec] &)
];


and replacing everything between the lines starting with ts2=... and finishing before DateListPlot, with

(* check for same dimensions *)
correctQ0 = SameQ @@ (Dimensions /@ {ts2["Path"], newdata});
If[
correctQ0,
{correctQ1, correctQ2} = Through[{
(* same timestamps and values for cases other than NA *)
And @@ Cases[#, {{t1_, v1_}, {t2_, v2_}} :> True /; t1 == t2 && v1 == v2 && v2 =!= "NA"] &,
(* same timestamps for cases with NA's *)
And @@ Cases[#, {{t1_, v1_}, {t2_, v2_}} :> True /; t1 == t2 && v2 =!= "NA"] &
}[Transpose[{ts2["Path"], newdata}]]],
correctQ1 = correctQ2 = False
];

correctQ = And @@ {correctQ0, correctQ1, correctQ2};


produces the following result: