Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

In the output of a large computer simulation(thousands of cpus for 3 days), I found that one data point is missing in the result. Is it possible to reconstruct the missing value using the remaining data?

Assumptions: The dynamics in the data is generally smooth and there is no shape feature in it.

Here is the data:

data={-0.0949456,-0.2232,-0.15012,0.0364052,0.0841621,-0.0920731,-0.273141,-0.183935,0.0656169,0.140269,-0.0843562,-0.332979,-0.228146,0.0969235,0.208076,-0.0696226,-0.402307,-0.284266,0.128555,0.287427,-0.0454046,-0.479687,-0.353367,0.158121,0.377113,-0.00921489,-0.56245,-0.43569,0.182807,0.474626,0.0409945,-0.646659,-0.530218,0.199767,0.576109,0.106226,-0.727344,-0.634338,0.206674,0.676612,0.185864,-0.799108,-0.743778,0.202287,0.770614,0.277269,-0.856969,-0.852921,0.18688,0.85286,0.,-0.897303,-0.955487,0.162245,0.919149,0.475145,-0.91851,-1.04548,0.131507,0.967006,0.568474,-0.921269,-1.11803,0.0983087,0.995824,0.649178,-0.908188,-1.17004,0.0660878,1.00669,0.711966,-0.883046,-1.20025,0.0374795,1.00158,0.753415,-0.849936,-1.20902,0.0140558,0.983158,0.772116,-0.8123,-1.19776,-0.00394007,0.953394,0.768557,-0.772323,-1.16867,-0.0170188,0.914086,0.744676,-0.731382,-1.12393,-0.0260316,0.866382,0.703393,-0.689571,-1.06621,-0.0322879,0.811196,0.648481,-0.646799,-0.997935,-0.0368642,0.74929,0.583615,-0.602475,-0.921481,-0.0407482,0.681664,0.513284,-0.555671,-0.839684,-0.0446284,0.609182,0.440653,-0.507031,-0.755086,-0.0489018,0.533683,0.369238,-0.456793,-0.669996,-0.0537497,0.457136,0.301312,-0.405981,-0.586799,-0.0589172,0.381381,0.238713,-0.355861,-0.507526,-0.064281,0.308536,0.182517,-0.30782,-0.433796,-0.0694734,0.240373,0.133182,-0.263169,-0.366831,-0.0741541,0.178336,0.0906948,-0.222976,-0.30744,-0.0780402,0.123434,0.0547048,-0.187986,-0.25604,-0.0809698,0.0762321,0.0246802,-0.158568,-0.212644,-0.0828802,0.0368173,0.};

Also from other output information, I can determined that the problem data is at the index 51, which is set to 0 in the above.

So is it possible to reconstruct that value?


Here is my try:

I use the wavelet transform to identify the abrupt changes in the data, and manually adjust the data value, until I get a relative smooth wavelet transform.

Manipulate[
 data1 = data;
 data1[[51]] = data[[51]] + Abs[Mean[{data[[50]], data[[52]]}]]*x;
 cwd = ContinuousWaveletTransform[data1, 
   GaborWavelet[10], {Automatic, 12}];
 values = Abs[cwd[All, "Values"]];
 ListDensityPlot[values, ColorFunction -> "TemperatureMap", 
  PlotRange -> {All, {0, 50}, {0, 0.1 Max[values]}}, 
  ClippingStyle -> Automatic, InterpolationOrder -> 0]
 ,
 {x, -10, 30, 1}
 ]

enter image description here

Form above I determined by eye that when x=17 the wavelet is smoothest. So the missing data I get is 0.377765. And here is a plot of the data and the wavelet transform after the fix

enter image description here enter image description here

Question:

  1. What are the general ways of reconstruct missing data?
  2. Are there other methods to reconstruct the missing data, without looking by eye?
share|improve this question
    
Related: Interpolating 2D data with missing values –  Rahul Narain Nov 23 '13 at 3:52
1  
Update: I tried the approach in that question, but it doesn't work too well thanks to the rapid oscillation in the data. Only with InterpolationOrder -> 100 does it get close to the expected result, and with even higher orders it becomes unstable. –  Rahul Narain Nov 23 '13 at 4:00
add comment

1 Answer 1

up vote 9 down vote accepted

I don't think there are "general methods". Normally interpolation or curve fitting can be used.

Let's see your particular problem. You have five distinct samplings:

ListLinePlot[data[[# ;; -1 ;; 5]] & /@ Range@5]

enter image description here

The first one shows the problem:

ListLinePlot[data[[1 ;; -1 ;; 5]]]

enter image description here

Let's see which point is the outlier:

First@Ordering[-Abs@Differences@data[[1 ;; -1 ;; 5]]]
(*
 11
*)

Let's calculate a "right" value

Mean[{data[[1 ;; -1 ;; 5]][[10]], data[[1 ;; -1 ;; 5]][[12]]}]
(*
 0.376207
*)

And now we replace it in the original sequence

ListLinePlot@ ReplacePart[data, Position[data, 0.][[1]] -> 0.37620699999999996`]

enter image description here

Edit

If you want a slightly better approximation, you could do something like:

pata = Transpose[{Range@Length@data, data}]; 
Interpolation[Join[pata[[1 ;; -1 ;; 5]][[5 ;; 10]], 
                   pata[[1 ;; -1 ;; 5]][[12 ;; 18]]]][51]

(*
0.375886
*)
share|improve this answer
1  
That's clever. How do you determine the number 5? What do you mean by "five distinct samplings"? –  xslittlegrass Nov 23 '13 at 0:00
    
@xslittlegrass Your sampling freq is almost five times the highest freq of the signal. You are sampling five times each period –  belisarius Nov 23 '13 at 0:06
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.