8
$\begingroup$

In the output of a large computer simulation(thousands of cpus for 3 days), I found that one data point is missing in the result. Is it possible to reconstruct the missing value using the remaining data?

Assumptions: The dynamics in the data is generally smooth and there is no shape feature in it.

Here is the data:

data={-0.0949456,-0.2232,-0.15012,0.0364052,0.0841621,-0.0920731,-0.273141,-0.183935,0.0656169,0.140269,-0.0843562,-0.332979,-0.228146,0.0969235,0.208076,-0.0696226,-0.402307,-0.284266,0.128555,0.287427,-0.0454046,-0.479687,-0.353367,0.158121,0.377113,-0.00921489,-0.56245,-0.43569,0.182807,0.474626,0.0409945,-0.646659,-0.530218,0.199767,0.576109,0.106226,-0.727344,-0.634338,0.206674,0.676612,0.185864,-0.799108,-0.743778,0.202287,0.770614,0.277269,-0.856969,-0.852921,0.18688,0.85286,0.,-0.897303,-0.955487,0.162245,0.919149,0.475145,-0.91851,-1.04548,0.131507,0.967006,0.568474,-0.921269,-1.11803,0.0983087,0.995824,0.649178,-0.908188,-1.17004,0.0660878,1.00669,0.711966,-0.883046,-1.20025,0.0374795,1.00158,0.753415,-0.849936,-1.20902,0.0140558,0.983158,0.772116,-0.8123,-1.19776,-0.00394007,0.953394,0.768557,-0.772323,-1.16867,-0.0170188,0.914086,0.744676,-0.731382,-1.12393,-0.0260316,0.866382,0.703393,-0.689571,-1.06621,-0.0322879,0.811196,0.648481,-0.646799,-0.997935,-0.0368642,0.74929,0.583615,-0.602475,-0.921481,-0.0407482,0.681664,0.513284,-0.555671,-0.839684,-0.0446284,0.609182,0.440653,-0.507031,-0.755086,-0.0489018,0.533683,0.369238,-0.456793,-0.669996,-0.0537497,0.457136,0.301312,-0.405981,-0.586799,-0.0589172,0.381381,0.238713,-0.355861,-0.507526,-0.064281,0.308536,0.182517,-0.30782,-0.433796,-0.0694734,0.240373,0.133182,-0.263169,-0.366831,-0.0741541,0.178336,0.0906948,-0.222976,-0.30744,-0.0780402,0.123434,0.0547048,-0.187986,-0.25604,-0.0809698,0.0762321,0.0246802,-0.158568,-0.212644,-0.0828802,0.0368173,0.};

Also from other output information, I can determined that the problem data is at the index 51, which is set to 0 in the above.

So is it possible to reconstruct that value?


Here is my try:

I use the wavelet transform to identify the abrupt changes in the data, and manually adjust the data value, until I get a relative smooth wavelet transform.

Manipulate[
 data1 = data;
 data1[[51]] = data[[51]] + Abs[Mean[{data[[50]], data[[52]]}]]*x;
 cwd = ContinuousWaveletTransform[data1, 
   GaborWavelet[10], {Automatic, 12}];
 values = Abs[cwd[All, "Values"]];
 ListDensityPlot[values, ColorFunction -> "TemperatureMap", 
  PlotRange -> {All, {0, 50}, {0, 0.1 Max[values]}}, 
  ClippingStyle -> Automatic, InterpolationOrder -> 0]
 ,
 {x, -10, 30, 1}
 ]

enter image description here

Form above I determined by eye that when x=17 the wavelet is smoothest. So the missing data I get is 0.377765. And here is a plot of the data and the wavelet transform after the fix

enter image description here enter image description here

Question:

  1. What are the general ways of reconstruct missing data?
  2. Are there other methods to reconstruct the missing data, without looking by eye?
$\endgroup$
3
  • $\begingroup$ Related: Interpolating 2D data with missing values $\endgroup$
    – user484
    Nov 23 '13 at 3:52
  • 1
    $\begingroup$ Update: I tried the approach in that question, but it doesn't work too well thanks to the rapid oscillation in the data. Only with InterpolationOrder -> 100 does it get close to the expected result, and with even higher orders it becomes unstable. $\endgroup$
    – user484
    Nov 23 '13 at 4:00
  • $\begingroup$ Very nice idea indeed to use animation for manual determination of "the best" value! $\endgroup$ Nov 23 '14 at 13:16
11
$\begingroup$

I don't think there are "general methods". Normally interpolation or curve fitting can be used.

Let's see your particular problem. You have five distinct samplings:

ListLinePlot[data[[# ;; -1 ;; 5]] & /@ Range@5]

enter image description here

The first one shows the problem:

ListLinePlot[data[[1 ;; -1 ;; 5]]]

enter image description here

Let's see which point is the outlier:

First@Ordering[-Abs@Differences@data[[1 ;; -1 ;; 5]]]
(*
 11
*)

Let's calculate a "right" value

Mean[{data[[1 ;; -1 ;; 5]][[10]], data[[1 ;; -1 ;; 5]][[12]]}]
(*
 0.376207
*)

And now we replace it in the original sequence

ListLinePlot@ ReplacePart[data, Position[data, 0.][[1]] -> 0.37620699999999996`]

enter image description here

Edit

If you want a slightly better approximation, you could do something like:

pata = Transpose[{Range@Length@data, data}]; 
Interpolation[Join[pata[[1 ;; -1 ;; 5]][[5 ;; 10]], 
                   pata[[1 ;; -1 ;; 5]][[12 ;; 18]]]][51]

(*
0.375886
*)
$\endgroup$
2
  • 1
    $\begingroup$ That's clever. How do you determine the number 5? What do you mean by "five distinct samplings"? $\endgroup$ Nov 23 '13 at 0:00
  • $\begingroup$ @xslittlegrass Your sampling freq is almost five times the highest freq of the signal. You are sampling five times each period $\endgroup$ Nov 23 '13 at 0:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.