Delete rows from a dataset

Question

I have a dataset built from a call to SemanticImport("/data.csv", Delimiters -> ",") with data.csv of the form:

Src, Dst, Size, Arrival, Delay, SrcClk, TrgClk, Nhops, Route, SrcErr, TrgErr
12, 22, 100, 1199145666.72, 10.483, 2107, 107, 12, 7774334, 0.002, 0.048
12, 22, 100, 1199145696.88, 10.42, ffff, 107, 12, 7774334, 0.002, 0.048
....

I would like to erase rows whose column called "SrcClk" has some bad values (ie, values different than "2107"). I've tried doing this with DeleteCases such as :

DeleteCases[d,  "SrcClk" != 2107]

with no success. DeleteCases expects a list, not a dataset so I guess it's the problem but I don't know how. While looking for an answer, I found many answers added a "&" at the end of their query but I don't know why.

Answer 1

You should make a dataset query that uses Select to extract those rows which satisfy the predicate #SrcClk == 2107 &, a pure function that select those rows for which it evaluates to True. In a dataset or an associations. This notation is documented in the last example under Examples in Function.

keys = {"Src", "Dst", "Size", "Arrival", "Delay", "SrcClk", "TrgClk", "Nhops", "Route", 
  "SrcErr", "TrgErr"};
vals = {{12, 22, 100, 1199145666.72, 10.483, 2107, 107, 12, 7774334, 0.002, 0.048},
        {12, 22, 100, 1199145696.88, 10.42, ffff, 107, 12, 7774334, 0.002, 0.048}};
ds = Dataset[AssociationThread /@ (Rule[keys, #] & /@ vals)]

ds[Select[#SrcClk == 2107 &]]

Answer 2

A couple of things to be aware of, ffff in your example won't be treated as a string unless you enclose it with "" otherwise it will be treated as a symbol simplistically a variable that hasn't had a value set. So looking at @m_goldbergs example some usual approaches won't work. Am I right in suspecting the ffff is hexadecimal?

ds[Select[#SrcClk != ffff &]]  (* won't work *)

as

Head[ds[[2, "SrcClk"]]]  (* give *symbol* *)

First we can update the symbol to text, one way to do this is

ds = ds[{2 -> (<|#, "SrcClk" -> "ffff"|> &)}]  (* we can update value directly *)  

(* or we can update its Head  from Symbol to String *)  

ds1 = ds[{2 -> (<|#, "SrcClk" -> Apply[String, #"SrcClk"]|> &)}] 

Also note that operations on a dataset are non-destructive until you update ds with "ds=" or create a new dataset ds1 with ds1=

A couple of other version of dataset selects I find useful

 ds[Select[#SrcClk == "ffff" && #Dst == 22 &]]   (* && is an AND operator *)
 ds[Select[#SrcClk == "ffff" || #Dst == 22 &]]  (* || is OR  *)

If we make the Dataset SrcClk column all of type of Head String then we can use wilds cards with StringMatchQ

ds1 = ds[{1 -> (<|#, "SrcClk" -> "2107"|> &)}]  
ds1[Select[StringMatchQ[#"SrcClk", "fff*"] &], {"Src", "Nhops", "SrcClk"}]  

In the last example we have also included the optional selection of only 3 of the dataset keys.

Answer 3

If you want to use DeleteCases, then the following works. I have borrowed the prior formulation to set up the dataset, and use @m_goldberg "keys" and "ds" variables. Thank you.

delL = ReplacePart[ConstantArray[_, Length[keys]], 6 -> s_ /; ToString[s] != "2107"]
delA = AssociationThread[keys -> delL]
DeleteCases[ds, delA]

A couple of comments: first the temporary variables are here to make it clearer. Not pretty but show the intermediate results. Second the magic number 6 for the ReplacePart could be done by indexing into the keys list, but it just makes things bushier.

Finally, the FullForm of ds shows that the type of SrcClk is AnyType, because the data for the column has both the number 2107 and the string ffff. The ToString in the Conditional forces this conversion.