# Cross checking lists and dropping points that do not belong

I have a list of data that I would like to cross examine with a "correct" list of values and pick out the correct values.

For example:

dataCorrect={{387.615, 25791.5}, {388.861, 25708.8}, {411.116, 24317.1}, {441.734, 22631.7}, {442.573, 22588.8}, {455.528, 21946.4}, {459.317, 21765.3}}

data={{387.593, 35.67}, {388.841, 17.369}, {404.644, 18.1776}, {411.065, 22.952}, {419.926, 20.136}, {435.764, 49.776}, {440.713, 22.916}, {442.976, 16.657}, {447.321, 20.922}, {450.017, 24.372}, {455.409, 982.36}, {459.191, 589.4}}


dataCorrect is {pos, energy} from literature, while data is {pos, intensity} but has false points, such as '{404.644, 18.1776}'.

My goal is to be able to use the dataCorrect positions with an integrated uncertainty (+/- 0.5 for example) to select points close to the true positions. Resulting in the following while dropping false points.

{data position, data intensity, matching energy}

dataFiltered={{387.593, 35.67, 25791.5}, {388.841, 17.369, 25708.8}, {411.065, 22.952, 24317.1}, {440.713, 22.916, 22631.7}, {442.976, 16.657, 22588.8},{455.409, 982.36, 21946.4}, {459.191, 589.4, 21765.3}}


Thank you

• Define "false points." Nov 6, 2018 at 18:11
• They are points that are taken from a large, noisy set of data. I use FindPeaks[importdata[[All, 2]], 5, .06, 15] which does a decent job but will select points that are not florescence, just noise. Nov 6, 2018 at 18:13
• You need a crisp, unambiguous mathematical definition. Otherwise there is no way we can help you. Nov 6, 2018 at 18:24
• There's no way to select points from one list that are within a certain range from point on another list -dropping those that aren't? Nov 6, 2018 at 18:39
• (t = #;Select[First/@data,t-.5<=#<=t+.5&])&/@(First/@dataCorrect) this will return the points you want. One point is out of */- 0.5 and thats why not displayed. rest is list manipulation Nov 6, 2018 at 18:40

## 3 Answers

Nearest should do a good job for large datasets.

{posCorrect, intCorrect} = Transpose[DeveloperToPackedArray[dataCorrect]];
{p, en} = Transpose[DeveloperToPackedArray[data]];
δ = 0.5;(*tolereance*)

idx = Nearest[posCorrect -> "Index", p, {1, δ}]


{{1}, {2}, {}, {3}, {}, {}, {}, {5}, {}, {}, {6}, {7}}

For each element if the list p, this finds the index of the closest element in pos of distance less than δ, if available.

goodidx = Flatten[Position[idx, _?(Length[#] > 0 &), 1]]


{1, 2, 4, 8, 11, 12}

These are the indices of the positions in p that seem to be okay.

dataFiltered = Transpose[{p[[goodidx]], en[[goodidx]], intCorrect[[Flatten[idx]]]}]


{{387.593, 35.67, 25791.5}, {388.841, 17.369, 25708.8}, {411.065, 22.952, 24317.1}, {442.976, 16.657, 22588.8}, {455.409, 982.36, 21946.4}, {459.191, 589.4, 21765.3}}

I use DeveloperToPackedArray to pack the input data for more efficient processing. With these small input datasets, that does not matter, though.

Join @@ Table[
If[Abs[j[[1]] - k[[1]]] <= 0.5, Append[k, j[[2]]]],
{j, dataCorrect},
{k, data}
] // DeleteCases[#, Null] &


{{387.593, 35.67, 25791.5}, {388.841, 17.369, 25708.8}, {411.065, 22.952, 24317.1}, {442.976, 16.657, 22588.8}, {455.409, 982.36, 21946.4}, {459.191, 589.4, 21765.3}}

And as was pointed out in the comments, the point {440.713, 22.916, 22631.7} does not meet the criterion of being within 0.5.

s=(t=#;Select[First/@data,t-.5<=#<=t+.5&])&/@(First/@dataCorrect);
Quiet@Table[
Join[data[[First@(# & @@
Position[data, First@s[[k]]])]], {dataCorrect[[k]][[2]]}], {k,
Length@dataCorrect}]
`