11
$\begingroup$

In version 11.0 (Win 8.1 Pro) a query on a large Dataset for a column that contains Missing fails most unceremoniously.

With

dsTest = AssociationThread[{"a", "b"}, 
     If[Last@# == 10000, {Missing["No Data"], Last@#}, #]] & /@ 
   Transpose@{RandomChoice[Alphabet[], 22000], Range[22000]} // 
  Dataset;

One row has column "a" set to Missing["No Data"]. Then

dsTest[All, "a"]

enter image description here

Bug, yes?

Work-around

A work-around is to remove the Missing instances from the Select.

dsTest[Select[Head[#"a"] =!= Missing &], "a"]

More work-arounds in comments/answers below.

Confirmed Bug - [CASE:3720717]

$\endgroup$
1
  • 2
    $\begingroup$ I reported it as a bug $\endgroup$
    – Jason B.
    Sep 21, 2016 at 16:41

2 Answers 2

8
$\begingroup$

Extended comment converted to a non-answer, nailing down what is required to trigger this bug:

Here's a simple Dataset with one key having all string values, another having integer values:

ds = Association @@@ ({"a" -> ToString@#, "b" -> #} & /@ Range[20]) // 
  ReplacePart[{1, 1} -> Missing[]] // Dataset

Mathematica graphics

If we query the integer values, then we see that the displayed form changes based on whether the number of rows we ask for is less than 9,

{ds[;; 8, "b"], ds[;; 9, "b"], ds[All, "b"]}

enter image description here

It switches from a column view to a grid-view when you query more than 8 elements. This is also where you have a problem with the "a" column, the Key with String values,

{ds[;; 8, "a"], ds[;; 9, "a"]}

enter image description here

A workaround is to set a global preference to use the old Dataset formatting,

Dataset`$UseNewDatasetFormatting = False;
{ds[;; 8, "a"], ds[;; 9, "a"]}

Mathematica graphics

$\endgroup$
1
  • 1
    $\begingroup$ Good work-around (+1). $\endgroup$
    – Edmund
    Sep 21, 2016 at 16:47
1
$\begingroup$

Too long for a comment: Another example narrowed down to a curious threshold:

vars = {"Var_A", "Var_B", "Var_C"};
assoc = AssociationMap[Random[] &] /@ Array[vars[[#2]] &, {9999, 3}];
assoc2 = Prepend[<|"Var_A" -> Random[], "Var_B" -> Random[], "Var_C" -> Missing[]|>]@assoc;
Predict[assoc -> "Var_C"]
Predict[assoc2 -> "Var_C"]

works fine but now change 9999 to 10 000 and Predict fails on assoc2 apparently confused by the Missing[] in the predicted variable.

(Confirmed as a bug a few weeks back)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.