1
$\begingroup$

I have a list of data that I would like to cross examine with a "correct" list of values and pick out the correct values.

For example:

dataCorrect={{387.615, 25791.5}, {388.861, 25708.8}, {411.116, 24317.1}, {441.734, 22631.7}, {442.573, 22588.8}, {455.528, 21946.4}, {459.317, 21765.3}}

data={{387.593, 35.67}, {388.841, 17.369}, {404.644, 18.1776}, {411.065, 22.952}, {419.926, 20.136}, {435.764, 49.776}, {440.713, 22.916}, {442.976, 16.657}, {447.321, 20.922}, {450.017, 24.372}, {455.409, 982.36}, {459.191, 589.4}} 

dataCorrect is {pos, energy} from literature, while data is {pos, intensity} but has false points, such as '{404.644, 18.1776}'.

My goal is to be able to use the dataCorrect positions with an integrated uncertainty (+/- 0.5 for example) to select points close to the true positions. Resulting in the following while dropping false points.

{data position, data intensity, matching energy}

dataFiltered={{387.593, 35.67, 25791.5}, {388.841, 17.369, 25708.8}, {411.065, 22.952, 24317.1}, {440.713, 22.916, 22631.7}, {442.976, 16.657, 22588.8},{455.409, 982.36, 21946.4}, {459.191, 589.4, 21765.3}}

Thank you

$\endgroup$
5
  • $\begingroup$ Define "false points." $\endgroup$ Nov 6, 2018 at 18:11
  • $\begingroup$ They are points that are taken from a large, noisy set of data. I use FindPeaks[importdata[[All, 2]], 5, .06, 15] which does a decent job but will select points that are not florescence, just noise. $\endgroup$
    – J. Joo
    Nov 6, 2018 at 18:13
  • 1
    $\begingroup$ You need a crisp, unambiguous mathematical definition. Otherwise there is no way we can help you. $\endgroup$ Nov 6, 2018 at 18:24
  • $\begingroup$ There's no way to select points from one list that are within a certain range from point on another list -dropping those that aren't? $\endgroup$
    – J. Joo
    Nov 6, 2018 at 18:39
  • $\begingroup$ (t = #;Select[First/@data,t-.5<=#<=t+.5&])&/@(First/@dataCorrect) this will return the points you want. One point is out of */- 0.5 and thats why not displayed. rest is list manipulation $\endgroup$
    – ZaMoC
    Nov 6, 2018 at 18:40

3 Answers 3

2
$\begingroup$

Nearest should do a good job for large datasets.

{posCorrect, intCorrect} = Transpose[Developer`ToPackedArray[dataCorrect]];
{p, en} = Transpose[Developer`ToPackedArray[data]];
δ = 0.5;(*tolereance*)

idx = Nearest[posCorrect -> "Index", p, {1, δ}]

{{1}, {2}, {}, {3}, {}, {}, {}, {5}, {}, {}, {6}, {7}}

For each element if the list p, this finds the index of the closest element in pos of distance less than δ, if available.

goodidx = Flatten[Position[idx, _?(Length[#] > 0 &), 1]]

{1, 2, 4, 8, 11, 12}

These are the indices of the positions in p that seem to be okay.

dataFiltered = Transpose[{p[[goodidx]], en[[goodidx]], intCorrect[[Flatten[idx]]]}]

{{387.593, 35.67, 25791.5}, {388.841, 17.369, 25708.8}, {411.065, 22.952, 24317.1}, {442.976, 16.657, 22588.8}, {455.409, 982.36, 21946.4}, {459.191, 589.4, 21765.3}}

I use Developer`ToPackedArray to pack the input data for more efficient processing. With these small input datasets, that does not matter, though.

$\endgroup$
0
$\begingroup$
Join @@ Table[
   If[Abs[j[[1]] - k[[1]]] <= 0.5, Append[k, j[[2]]]],
   {j, dataCorrect},
   {k, data}
   ] // DeleteCases[#, Null] &

{{387.593, 35.67, 25791.5}, {388.841, 17.369, 25708.8}, {411.065, 22.952, 24317.1}, {442.976, 16.657, 22588.8}, {455.409, 982.36, 21946.4}, {459.191, 589.4, 21765.3}}

And as was pointed out in the comments, the point {440.713, 22.916, 22631.7} does not meet the criterion of being within 0.5.

$\endgroup$
0
$\begingroup$
s=(t=#;Select[First/@data,t-.5<=#<=t+.5&])&/@(First/@dataCorrect);
Quiet@Table[
Join[data[[First@(# & @@ 
    Position[data, First@s[[k]]])]], {dataCorrect[[k]][[2]]}], {k,
Length@dataCorrect}]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.