I have a dataset built from a call to SemanticImport("/data.csv", Delimiters -> ",") with data.csv of the form:

Src, Dst, Size, Arrival, Delay, SrcClk, TrgClk, Nhops, Route, SrcErr, TrgErr
12, 22, 100, 1199145666.72, 10.483, 2107, 107, 12, 7774334, 0.002, 0.048
12, 22, 100, 1199145696.88, 10.42, ffff, 107, 12, 7774334, 0.002, 0.048

I would like to erase rows whose column called "SrcClk" has some bad values (ie, values different than "2107"). I've tried doing this with DeleteCases such as :

DeleteCases[d,  "SrcClk" != 2107]

with no success. DeleteCases expects a list, not a dataset so I guess it's the problem but I don't know how. While looking for an answer, I found many answers added a "&" at the end of their query but I don't know why.

  • $\begingroup$ Welcome to Mathematica.SE! I suggest that: 1) You take the introductory Tour now! 2) When you see good questions and answers, vote them up by clicking the gray triangles, because the credibility of the system is based on the reputation gained by users sharing their knowledge. Also, please remember to accept the answer, if any, that solves your problem, by clicking the checkmark sign! 3) As you receive help, try to give it too, by answering questions in your area of expertise. $\endgroup$ – bbgodfrey Mar 4 '15 at 14:47
  • $\begingroup$ Please add a small sample of your data to the question. $\endgroup$ – bbgodfrey Mar 4 '15 at 14:48
  • $\begingroup$ @mattator It's not mentioned particularly well on the Dataset help pages but take a look the Query function of which dataset[some function] is a special case of. Also have a look at Slot which #name is a shorthand for which will explain the &. $\endgroup$ – Gordon Coale Mar 5 '15 at 7:04
  • 1
    $\begingroup$ I think your problem is that you are intermixing syntax for patterns and functions. DeleteCases expects a pattern, but you do provide (half of) a function (the other half is the missing &). You probably really should look into the documentation for DeleteCases, Select and Function to understand why and when a & is mandatory. On the other hand to delete from a Dataset you'd typically use the new Query syntax specifically made for them: d[Select[(#SrcClk=!=2107)&]], which you'll find in the documentation for Dataset... $\endgroup$ – Albert Retey Mar 5 '15 at 14:10

You should make a dataset query that uses Select to extract those rows which satisfy the predicate #SrcClk == 2107 &, a pure function that select those rows for which it evaluates to True. In a dataset or an associations. This notation is documented in the last example under Examples in Function.

keys = {"Src", "Dst", "Size", "Arrival", "Delay", "SrcClk", "TrgClk", "Nhops", "Route", 
  "SrcErr", "TrgErr"};
vals = {{12, 22, 100, 1199145666.72, 10.483, 2107, 107, 12, 7774334, 0.002, 0.048},
        {12, 22, 100, 1199145696.88, 10.42, ffff, 107, 12, 7774334, 0.002, 0.048}};
ds = Dataset[AssociationThread /@ (Rule[keys, #] & /@ vals)]


ds[Select[#SrcClk == 2107 &]]



A couple of things to be aware of, ffff in your example won't be treated as a string unless you enclose it with "" otherwise it will be treated as a symbol simplistically a variable that hasn't had a value set. So looking at @m_goldbergs example some usual approaches won't work. Am I right in suspecting the ffff is hexadecimal?

ds[Select[#SrcClk != ffff &]]  (* won't work *)


Head[ds[[2, "SrcClk"]]]  (* give *symbol* *)

First we can update the symbol to text, one way to do this is

ds = ds[{2 -> (<|#, "SrcClk" -> "ffff"|> &)}]  (* we can update value directly *)  

(* or we can update its Head  from Symbol to String *)  

ds1 = ds[{2 -> (<|#, "SrcClk" -> Apply[String, #"SrcClk"]|> &)}] 

Also note that operations on a dataset are non-destructive until you update ds with "ds=" or create a new dataset ds1 with ds1=

A couple of other version of dataset selects I find useful

 ds[Select[#SrcClk == "ffff" && #Dst == 22 &]]   (* && is an AND operator *)
 ds[Select[#SrcClk == "ffff" || #Dst == 22 &]]  (* || is OR  *)

If we make the Dataset SrcClk column all of type of Head String then we can use wilds cards with StringMatchQ

ds1 = ds[{1 -> (<|#, "SrcClk" -> "2107"|> &)}]  
ds1[Select[StringMatchQ[#"SrcClk", "fff*"] &], {"Src", "Nhops", "SrcClk"}]  

In the last example we have also included the optional selection of only 3 of the dataset keys.

  • $\begingroup$ There was indeed a problem with the types, thanks $\endgroup$ – mattator Apr 29 '15 at 18:25

If you want to use DeleteCases, then the following works. I have borrowed the prior formulation to set up the dataset, and use @m_goldberg "keys" and "ds" variables. Thank you.

delL = ReplacePart[ConstantArray[_, Length[keys]], 6 -> s_ /; ToString[s] != "2107"]
delA = AssociationThread[keys -> delL]
DeleteCases[ds, delA]

A couple of comments: first the temporary variables are here to make it clearer. Not pretty but show the intermediate results. Second the magic number 6 for the ReplacePart could be done by indexing into the keys list, but it just makes things bushier.

Finally, the FullForm of ds shows that the type of SrcClk is AnyType, because the data for the column has both the number 2107 and the string ffff. The ToString in the Conditional forces this conversion.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.