2
$\begingroup$

I'm trying to analyse data with Mathematica. I have a big array with various columns representing timestamps and then lots of data. I have some code that will produce a list of timestamps when the data is bad and should be deleted.

I would like to be able to take any new set of data and identify the position of any timestamps that are in the list of bad data and should be deleted.

To put it bit more precisely, given list A and list B is there way to output the position where of any member of list A occurs in list B.

I have some terrible code that does the job, but I'm sure there is a nicer way to do this with a function that just has two lists as the input. In this code datatimelist is the time stamps of the new data, and glitchtimelist is the previously identified timestamps of bad data from a different experiment.

datatimelist = thedata[[All, 3]]
match[x_] := MemberQ[glitchtimelist, x];
badpointlist = Position[Map[match, datatimelist], True];
datadeglitched = Delete[thedata, badpointlist];    

Thanks Fred.

$\endgroup$
  • $\begingroup$ You might want to look at Pick[]. $\endgroup$ – J. M. is away Jun 8 '15 at 14:30
  • $\begingroup$ Post some actual data... or dummy data if you want a quality answer. I recommend using Complement or Intersection to find what sets of elements you want, then Pick or Delete elements based on the result. $\endgroup$ – Histograms Jun 8 '15 at 14:31
  • $\begingroup$ Welcome to Mathematica.SE! I hope you will become a regular contributor. To get started, 1) take the introductory Tour now, 2) when you see good questions and answers, vote them up by clicking the gray triangles, because the credibility of the system is based on the reputation gained by users sharing their knowledge, 3) remember to accept the answer, if any, that solves your problem, by clicking the checkmark sign, and 4) give help too, by answering questions in your areas of expertise. $\endgroup$ – bbgodfrey Jun 8 '15 at 14:32
  • $\begingroup$ Some dummy data would be glitchtimelist={2,3} for the glitches and datatimelist={{1,2.31231},{2,2.11241},{3,2.55345},{4,2,23452.6},{5,2.22352} for the timestamps and data. I want to know the position of the points {2,2.11241} and {3,2.55345} since the have a timestamp that's in glitchtimelist. I would also love it in the form of a function that can have two lists as the input $\endgroup$ – Fred Jun 8 '15 at 15:12
  • 2
    $\begingroup$ Pick[thedata, Not@*match /@ datatimelist] $\endgroup$ – C. E. Jun 8 '15 at 15:58
3
$\begingroup$

Let's compare two flavors based on Pick (slightly modified version from Pickett) and Select. I will assume that data is in {timestamp, value} format and glitches data is just timestamp. Both functions return filtered dataset that has no glitches datapoints. I will start a fresh kernel.

glitchtimelist = {2, 3};
datatimelist = {{1, 2.31231}, {2, 2.11241}, {3, 2.55345}, {4, 2, 
    23452.6}, {5, 2.22352}};

f[thedata_, glitchtimelist_] := 
  Select[thedata, Not@MemberQ[glitchtimelist, #[[1]]] &];
g[thedata_, glitchtimelist_] := 
  Pick[thedata, Not@*(MemberQ[glitchtimelist, #[[1]]] &) /@ thedata];
f[datatimelist, glitchtimelist]
g[datatimelist, glitchtimelist]

Both give {{1, 2.31231}, {4, 2, 23452.6}, {5, 2.22352}}. Now we will measure memory consumption for big dataset having a million points with one thousand of glitches.

SeedRandom[1234];

data = {#, RandomReal[]} & /@ Range[1000000];
glitch = RandomSample[Range[1000000], 1000];

First@AbsoluteTiming@g[data, glitch]
MaxMemoryUsed[]

My result is 70 seconds and 363Mb of memory. Now we quit the kernel and run the same test for f. The result is 67 seconds and 316Mb of memory. So Pick is slightly less memory efficient which probably won't matter much.

$\endgroup$
  • $\begingroup$ #[[1]]& is a pure function that takes the first element of a list. See more here reference.wolfram.com/language/ref/Slot.html and reference.wolfram.com/language/tutorial/PureFunctions.html $\endgroup$ – BlacKow Jun 8 '15 at 18:47
  • $\begingroup$ Great, that works really well, thanks. So, Select is applied to the data set with the criteria that if the entry is not part of glitchtimelist it is selected. Could you explain to me the #[[1]]] & part of the function ? $\endgroup$ – Fred Jun 8 '15 at 18:53
  • $\begingroup$ The second argument of Select is a function. This function is applied to all elements of the list that sits in the first argument. If it returns True the element is selected. Since your data consists of pairs {timestamp,value} you need to extract timestamp, i.e. take the first element. aaa[[1]] gives the first element of aaa. #[[1]]& is a function that gives the first element. Just another way (but often very convenient!) to have function without naming argument. $\endgroup$ – BlacKow Jun 8 '15 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.