# Interpolating missing data

I have a data set that contains data of the form (x0, y0, f1, f2, i1, i2, i3). The (x0, y0) are the coordinates, while the values f1 and f2 are real numbers (i1, i2, i3 correspond to some integers which are used as indices). The data can be downloaded here.

Now I plot the (x0, y0) coordinates of the data with i2 = 4, where each point is colored according to the value of f1.

As you can see, there are missing points. The data set with all the missing points can be found here

Now, how can I use the original data with the f1 and f2 values, so as to interpolate and predict the f1 and f2 values of the missing points? Any suggestions?

SetDirectory[NotebookDirectory[]];
data = Import["basins_(L4).out.txt", "Table"];


interpolate $$f_1$$ and $$f_2$$: linear interpolation on irregular grid,

F1 = Interpolation[{{#[[1]], #[[2]]}, #[[3]]} & /@
Select[data, #[[6]] == 4 &], InterpolationOrder -> 1];
F2 = Interpolation[{{#[[1]], #[[2]]}, #[[4]]} & /@
Select[data, #[[6]] == 4 &], InterpolationOrder -> 1];


evaluate $$f_1$$ and $$f_2$$ on the entire grid (interpolate missing data):

T1 = Table[{x,y, F1[x,y]}, {x,Union[data[[All, 1]]]}, {y,Union[data[[All,2]]]}];
T2 = Table[{x,y, F2[x,y]}, {x,Union[data[[All, 1]]]}, {y,Union[data[[All,2]]]}];


plots of the interpolated functions in the style of @kickert's solution:

ListPointPlot3D[Join @@ T1]


ListPointPlot3D[Join @@ T2]


The Predict function can provide you the information you need.

Start by importing your data into Mathematica. For me, it was easiest to change the file extensions to .txt and use SemanticImport.

rawdata = SemanticImport["basins_(L4).txt"] // Normal;
missing = SemanticImport["data_LGs.txt"] // Normal;


Then pull out the subset that with i2=4.

subset = Select[rawdata, #[[6]] == 4 &]


You can now thread your (x0, y0) values to the f1 values:

f1aidata = Thread[subset[[All, 1 ;; 2]] -> subset[[All, 3]]]


At this point you have some choices to make around the Method and Performance Goals you use for the Predict function. We could go deep in the weeds on this, but I created some training and test data and ran through all the options and found GradientBoostedTrees was the best compromise between quality and computational time.

f1predictor =
PerformanceGoal -> "Quality"]


With the Predictor you just created, you can run the missing data through it.

f1outputs = f1predictor[#] & /@ missing;


Then combine the inputs and outputs and Join the lists

f1missingresults = Append[Transpose[missing], f1outputs] // Transpose;
combinedresults = Join[subset[[All, 1 ;; 3]], f1missingresults];


Using a ListDensityPlot, you get this:

ListDensityPlot[combinedresults, ColorFunction -> "TemperatureMap"]


Looking at the ListPointPlot3D you can see it isn't perfect, but it is very close.

ListPointPlot3D[combinedresults]


If you want to use this for f2, then follow the same process pulling your data from subset[[All,{1,2,4}]] and creating a new predictor

• GradientBoostedTrees seems to get much worse results than a simple linear interpolation: with the latter, you cannot see any residual structure in the ListPointPlot3D of the result. Jun 28 '19 at 15:33
• I actually found that the GaussianProcess method was more accurate, but running the full data set through the Predict function was prohibitively time consuming. The interpolation approach is much faster and cleaner. Jun 28 '19 at 15:41
• I've been doing a lot of machine learning work recently... When you have a hammer in your hand, everything looks like a nail. ;-) Jun 28 '19 at 15:48
• No worries, this is certainly a very interesting method for when the data are a bit less regular and ML can show its strengths better. Jun 28 '19 at 17:02
• (+1) For imputing missing values with Decision Trees (DT's) -- DT's can be very good for imputation. (And, yes, there are better solutions for this case...) Jun 28 '19 at 22:47