0
$\begingroup$

consider this Dataset:

dataset = Dataset[{
     <|"a" -> 1, "b" -> "x", "c" -> {1}|>
   , <|"a" -> 2, "b" -> "y",  "c" -> {2, 3}|>
   , <|"a" -> 3, "b" -> "z", "c" -> {3}|>
   , <|"a" -> 4, "b" -> "x", "c" -> Missing["Unmatched"]|>
   , <|"a" -> 5, "b" -> Missing["Unmatched"], "c" -> {9} |>
   , <|"a" -> 6, "b" -> "z", "c" -> {}|>
}]

I want to delete all rows that contain Missing["Unmatched"].

Question:

What is an elegant way of performing this operation?

My real dataset is 300 MB and has 50 columns. The Missing["Unmatched"] values can occur in any column.

Example of a large dataset

bigDataset = Dataset@Table[
                           <|"a" -> j
                           , "b" -> RandomChoice[{RandomInteger[{0, 9}], Missing["Unmatched"]}]
                           , "c" -> RandomChoice[{RandomInteger[{0, 9}], Missing["Unmatched"]}]
                         |>
                         , {j, 1, 1000000, 1}];
$\endgroup$
2
  • 1
    $\begingroup$ It will be difficult to give you accurate performance comparisons against your file if we don’t have an example of the same size/complexity to test possible solutions upon; though, I think you may have realized this, but it is worth mentioning in any case. $\endgroup$ May 30 at 16:16
  • $\begingroup$ @CATrevillian good point. I have added an example of a large dataset similar to my question. $\endgroup$ May 30 at 20:11
2
$\begingroup$

Testing on your sample data dataset:

r1 = DeleteMissing[dataset, 1, 1]

Takes 0.000530041 second

r2 = dataset[Select[Not@AnyTrue[#, MissingQ] &]]

Takes 0.000412052 second

r3 = dataset[Select[Not@MatchQ[#, KeyValuePattern[{_ -> _?MissingQ}]] &]]

Takes 0.000428904 second

All have the same result:

r1 == r2 == r3

(*Out: True *)

enter image description here

$\endgroup$
1
$\begingroup$

one way to do this is:

Dataset@DeleteCases[
                    Normal[dataset],
                    <|___, _ -> Missing["Unmatched"], ___|>
                 ];

which takes $2.32$ seconds on bigDataset (in my question)

Benchmarking @BenIzd 's methods and my ownenter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.