2
$\begingroup$

I am importing data for creating a machine learning classifier and at the moment i am cleaning the data. There are missing values within the data and are marked NA

missingData = {{2, 4, 0.986`,RGBColor[0.7931811599535044, 0.9794035526034883, 
0.1736111262263147]}, {1, 5, 0.0701`,RGBColor[0.4966346762097671, 0.42506700252493146, 
0.6020480735371379]}, {2, 3, 0.356`,RGBColor[0.17005048612802431, 0.32340985960079616, 
0.5088057001465967]}, {1, 5, 0.859`,RGBColor[0.26079904402442433, 0.6645405387012917, 
0.626762001873512]}, {1, 7, 0.87`,RGBColor[0.35006141637857935, 0.241784130785629, 
0.8287210560502678]}, {1, 5, 0.372`,RGBColor[0.6040242604212316, 0.2495041618176539, 
0.1612746936530718]}, {2, "NA", 0.356`,RGBColor[0.5812714623512671, 0.2969407447142949, 
0.49572510310649687]}, {1, 3, 0.751`,RGBColor[0.6124614477104853, 0.378535780066783, 
0.878636444824388]}, {1, 1, 0.157`,RGBColor[0.664181976745267, 0.25405278277799814, 
0.12343422551954197]}, {1, 1, "NA",RGBColor[0.5918739987770754, 0.7079781022288523, 
0.7994674398394699]}, {2, 3, 0.685`, "NA"}, {2, 5, 0.458`,RGBColor[0.8929266436889802, 0.6764412122657568, 
0.1876179631453354]}, {2, 4, 0.0106`,RGBColor[0.5746966560731546, 0.5532290683267522, 
0.6433151858365447]}, {1, 0, 0.697`,RGBColor[0.4036028221713692, 0.6499596806079162, 
0.1976339227708881]}, {2, 6, 0.0239`,RGBColor[0.2242880379356622, 0.11173047800711289, 
0.0796914387814045]}}

I am trying to use hot deck imputation(HDI) to replace the missing values

copyOfMissingData=missingData

The Problem I am able to use HDI to replace only one of the variables with the follwing input Listing the samples with missing data

missing = Cases[copyOfMissingData, {___, "NA"}]

List the good samples (with no missing data) :

goodSamples = DeleteCases[copyOfMissingData, {___, "NA",___}]

Find the sample that is nearest to the sample "missing", based on feature values in columns 1, 2 and 3 :

Nearest[goodSamples[[All, {1, 2, 3}]], First[missing][[{1, 2,3}]]]

Set up Nearest to return the color (data from column 4) for the sample that is nearest to "missing" for values in columns 1, 2 and 3 :

samples = #[[{1, 2, 3}]] -> #[[4]] & /@ goodSamples

replacement = Nearest[samples, First[missing][[{1, 2, 3}]]]

Replace the missing data with the feature value from the nearest sample :

copyOfMissingData[[All, 4]] /."NA" -> RGBColor[0.17005048612802431`, 0.32340985960079616`,0.5088057001465967]

However this works only for replacing one of the missing variables. when i try to use this code to replace all missing variables, i encounter the following. The only change i make is in the missing input, where i include the rest of the variables

missing = Cases[copyOfMissingData, {___, "NA", ___}]
goodSamples = DeleteCases[copyOfMissingData, {___, "NA", ___}]

Find the sample that is nearest to the sample "missing", based on feature values in columns 1, 2 and 3

Nearest[goodSamples[[All, {1, 2, 3}]], First[missing][[{1, 2,3}]]]

This is where i encounter the following error message

Nearest::neard: The default distance function does not give a real numeric distance when applied to the point pair {2,NA,0.356} and {2,4,0.986}.

What im aiming to do is replace all of the missing variables in this dataset using hot deck imputation. Any help would be appreciated

$\endgroup$
  • 2
    $\begingroup$ Rather than using HDI have you considered using LearnDistribution and SynthesizeMissingValues see this for examples. $\endgroup$ – Rohit Namjoshi Feb 28 at 15:10
  • $\begingroup$ Hello Rohit, thank you for reply. I was able to use LearnDistribution successfully, thanks for the tip. At the moment I am trying to learn different methods of cleaning data so any help would be appreciated $\endgroup$ – Luke4737 Feb 28 at 17:11
2
$\begingroup$
noMissing = Select[FreeQ["NA"]][missingData];

For versions before 12.0,

ClearAll[dist, nearestF, hdi]
dist[u : {__?NumericQ}, v : {__?NumericQ}] := EuclideanDistance[u, v]
dist[{u__?NumericQ, c1_RGBColor}, {v__?NumericQ, c2_RGBColor}] :=  dist[{u}, {v}] + 
    ColorDistance[c1, c2]

nearestF[i_] := First@Nearest[Drop[noMissing, None, {i}] -> "Index", #, 1, 
    DistanceFunction -> dist] &

hdi = Map[# /. p : {__} /; Not[FreeQ[p, "NA"]] :> 
   Module[{pos = First[PositionIndex[p]@"NA"]},
     Replace[p, "NA" -> noMissing[[nearestF[pos][Drop[p, {pos}]], pos]], All]] &];

Row[Panel[Grid[#]] & /@ {missingData, hdi @ missingData}, Spacer[10]]

enter image description here

For version 12.0, TIL from @Rohit's comment the function SynthesizeMissingValues.

| improve this answer | |
$\endgroup$
  • $\begingroup$ OMG, TIL @kglr does not know every function in WL. :-( $\endgroup$ – Rohit Namjoshi Feb 28 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.