I have a data set that contains data of the form (x0, y0, f1, f2, i1, i2, i3). The (x0, y0) are the coordinates, while the values f1 and f2 are real numbers (i1, i2, i3 correspond to some integers which are used as indices). The data can be downloaded here.

Now I plot the (x0, y0) coordinates of the data with i2 = 4, where each point is colored according to the value of f1.

enter image description here

As you can see, there are missing points. The data set with all the missing points can be found here

enter image description here

Now, how can I use the original data with the f1 and f2 values, so as to interpolate and predict the f1 and f2 values of the missing points? Any suggestions?


2 Answers 2


read the data:

data = Import["basins_(L4).out.txt", "Table"];

interpolate $f_1$ and $f_2$: linear interpolation on irregular grid,

F1 = Interpolation[{{#[[1]], #[[2]]}, #[[3]]} & /@ 
       Select[data, #[[6]] == 4 &], InterpolationOrder -> 1];
F2 = Interpolation[{{#[[1]], #[[2]]}, #[[4]]} & /@ 
       Select[data, #[[6]] == 4 &], InterpolationOrder -> 1];

evaluate $f_1$ and $f_2$ on the entire grid (interpolate missing data):

T1 = Table[{x,y, F1[x,y]}, {x,Union[data[[All, 1]]]}, {y,Union[data[[All,2]]]}];
T2 = Table[{x,y, F2[x,y]}, {x,Union[data[[All, 1]]]}, {y,Union[data[[All,2]]]}];

plots of the interpolated functions in the style of @kickert's solution:

ListPointPlot3D[Join @@ T1]

enter image description here

ListPointPlot3D[Join @@ T2]

enter image description here


The Predict function can provide you the information you need.

Start by importing your data into Mathematica. For me, it was easiest to change the file extensions to .txt and use SemanticImport.

rawdata = SemanticImport["basins_(L4).txt"] // Normal;
missing = SemanticImport["data_LGs.txt"] // Normal;

Then pull out the subset that with i2=4.

subset = Select[rawdata, #[[6]] == 4 &]

You can now thread your (x0, y0) values to the f1 values:

f1aidata = Thread[subset[[All, 1 ;; 2]] -> subset[[All, 3]]]

At this point you have some choices to make around the Method and Performance Goals you use for the Predict function. We could go deep in the weeds on this, but I created some training and test data and ran through all the options and found GradientBoostedTrees was the best compromise between quality and computational time.

f1predictor = 
 Predict[f1aidata, Method -> "GradientBoostedTrees", 
  PerformanceGoal -> "Quality"]

With the Predictor you just created, you can run the missing data through it.

f1outputs = f1predictor[#] & /@ missing;

Then combine the inputs and outputs and Join the lists

f1missingresults = Append[Transpose[missing], f1outputs] // Transpose;
combinedresults = Join[subset[[All, 1 ;; 3]], f1missingresults];

Using a ListDensityPlot, you get this:

ListDensityPlot[combinedresults, ColorFunction -> "TemperatureMap"]

enter image description here

Looking at the ListPointPlot3D you can see it isn't perfect, but it is very close.


enter image description here

If you want to use this for f2, then follow the same process pulling your data from subset[[All,{1,2,4}]] and creating a new predictor

  • $\begingroup$ GradientBoostedTrees seems to get much worse results than a simple linear interpolation: with the latter, you cannot see any residual structure in the ListPointPlot3D of the result. $\endgroup$
    – Roman
    Jun 28, 2019 at 15:33
  • $\begingroup$ I actually found that the GaussianProcess method was more accurate, but running the full data set through the Predict function was prohibitively time consuming. The interpolation approach is much faster and cleaner. $\endgroup$
    – kickert
    Jun 28, 2019 at 15:41
  • $\begingroup$ I've been doing a lot of machine learning work recently... When you have a hammer in your hand, everything looks like a nail. ;-) $\endgroup$
    – kickert
    Jun 28, 2019 at 15:48
  • $\begingroup$ No worries, this is certainly a very interesting method for when the data are a bit less regular and ML can show its strengths better. $\endgroup$
    – Roman
    Jun 28, 2019 at 17:02
  • $\begingroup$ (+1) For imputing missing values with Decision Trees (DT's) -- DT's can be very good for imputation. (And, yes, there are better solutions for this case...) $\endgroup$ Jun 28, 2019 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.