2
$\begingroup$

I'm trying to analyse data with Mathematica. I have a big array with various columns representing timestamps and then lots of data. I have some code that will produce a list of timestamps when the data is bad and should be deleted.

I would like to be able to take any new set of data and identify the position of any timestamps that are in the list of bad data and should be deleted.

To put it bit more precisely, given list A and list B is there way to output the position where of any member of list A occurs in list B.

I have some terrible code that does the job, but I'm sure there is a nicer way to do this with a function that just has two lists as the input. In this code datatimelist is the time stamps of the new data, and glitchtimelist is the previously identified timestamps of bad data from a different experiment.

datatimelist = thedata[[All, 3]]
match[x_] := MemberQ[glitchtimelist, x];
badpointlist = Position[Map[match, datatimelist], True];
datadeglitched = Delete[thedata, badpointlist];

Thanks Fred.

Improve this question
$\endgroup$
7
  • $\begingroup$ You might want to look at Pick[]. $\endgroup$
    – J. M.'s eventual burnout ♦
    Jun 8, 2015 at 14:30
  • $\begingroup$ Post some actual data... or dummy data if you want a quality answer. I recommend using Complement or Intersection to find what sets of elements you want, then Pick or Delete elements based on the result. $\endgroup$
    – Histograms
    Jun 8, 2015 at 14:31
  • $\begingroup$ Welcome to Mathematica.SE! I hope you will become a regular contributor. To get started, 1) take the introductory Tour now, 2) when you see good questions and answers, vote them up by clicking the gray triangles, because the credibility of the system is based on the reputation gained by users sharing their knowledge, 3) remember to accept the answer, if any, that solves your problem, by clicking the checkmark sign, and 4) give help too, by answering questions in your areas of expertise. $\endgroup$
    – bbgodfrey
    Jun 8, 2015 at 14:32
  • $\begingroup$ Some dummy data would be glitchtimelist={2,3} for the glitches and datatimelist={{1,2.31231},{2,2.11241},{3,2.55345},{4,2,23452.6},{5,2.22352} for the timestamps and data. I want to know the position of the points {2,2.11241} and {3,2.55345} since the have a timestamp that's in glitchtimelist. I would also love it in the form of a function that can have two lists as the input $\endgroup$
    – Fred
    Jun 8, 2015 at 15:12
  • 2
    $\begingroup$ Pick[thedata, Not@*match /@ datatimelist] $\endgroup$
    – C. E.
    Jun 8, 2015 at 15:58

1 Answer 1

Reset to default
3
$\begingroup$

Let's compare two flavors based on Pick (slightly modified version from Pickett) and Select. I will assume that data is in {timestamp, value} format and glitches data is just timestamp. Both functions return filtered dataset that has no glitches datapoints. I will start a fresh kernel.

glitchtimelist = {2, 3};
datatimelist = {{1, 2.31231}, {2, 2.11241}, {3, 2.55345}, {4, 2, 
    23452.6}, {5, 2.22352}};

f[thedata_, glitchtimelist_] := 
  Select[thedata, Not@MemberQ[glitchtimelist, #[[1]]] &];
g[thedata_, glitchtimelist_] := 
  Pick[thedata, Not@*(MemberQ[glitchtimelist, #[[1]]] &) /@ thedata];
f[datatimelist, glitchtimelist]
g[datatimelist, glitchtimelist]

Both give {{1, 2.31231}, {4, 2, 23452.6}, {5, 2.22352}}. Now we will measure memory consumption for big dataset having a million points with one thousand of glitches.

SeedRandom[1234];

data = {#, RandomReal[]} & /@ Range[1000000];
glitch = RandomSample[Range[1000000], 1000];

First@AbsoluteTiming@g[data, glitch]
MaxMemoryUsed[]

My result is 70 seconds and 363Mb of memory. Now we quit the kernel and run the same test for f. The result is 67 seconds and 316Mb of memory. So Pick is slightly less memory efficient which probably won't matter much.

Improve this answer
$\endgroup$
3
  • $\begingroup$ #[[1]]& is a pure function that takes the first element of a list. See more here reference.wolfram.com/language/ref/Slot.html and reference.wolfram.com/language/tutorial/PureFunctions.html $\endgroup$
    – BlacKow
    Jun 8, 2015 at 18:47
  • $\begingroup$ Great, that works really well, thanks. So, Select is applied to the data set with the criteria that if the entry is not part of glitchtimelist it is selected. Could you explain to me the #[[1]]] & part of the function ? $\endgroup$
    – Fred
    Jun 8, 2015 at 18:53
  • $\begingroup$ The second argument of Select is a function. This function is applied to all elements of the list that sits in the first argument. If it returns True the element is selected. Since your data consists of pairs {timestamp,value} you need to extract timestamp, i.e. take the first element. aaa[[1]] gives the first element of aaa. #[[1]]& is a function that gives the first element. Just another way (but often very convenient!) to have function without naming argument. $\endgroup$
    – BlacKow
    Jun 8, 2015 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.