I have the following Dataset:

ds = Dataset@*Map[AssociationThread[{"ID", "VAL1", "VAL2", "VAL3", "Type"} -> #] &]@
    {{"C09", 18.5, 18.5, 18.5, "FE"}, {"C11", 21, 21, 21, "FE"}, 
    {"C10", 32, 32, 32, "FE"}, {"C14", 18.5, 18.6, 18.5, "FE"},
    {"C21", 34, 34, 34, "FE"}, {"C22", 21, 21, 21, "FE"}, 
    {"C27", 11.5, 23, 10, "FE"}}

enter image description here

And I'm trying to change the "Type" value (from "FE" to "NONE") under the condition, that VAL1==VAL2==VAL3

My guess was:

     newds = ds[All,If[Select[#"VAL1" == #"VAL2" == #"VAL3" &], <|#,"Type" ->"NONE"|>, #] &]

enter image description here

I tried a lot and checked different ways but it couldn't succeed. Any ideas how to get this done?

  • 1
    $\begingroup$ ds[All, If[#"VAL1" == #"VAL2" == #"VAL3", <|#, "Type" -> "None"|>, #] &] this works but gives errors. I don't know why $\endgroup$
    – Kuba
    Feb 16 '15 at 13:06
  • $\begingroup$ Closely related: (59799) $\endgroup$
    – Mr.Wizard
    Feb 16 '15 at 19:00

The problem with the query proposed in the question is that it is attempting to apply Select to each row. As @Kuba points out in a comment, the use of Select is unnecessary. The query can be expressed like this:

ds[All, If[#"VAL1" == #"VAL2" == #"VAL3", <|#, "Type" -> "None"|>, #] &]

dataset screenshot

The same result can be obtained by an alternate formulation:

ds[All, <| #, "Type" -> If[#"VAL1" == #"VAL2" == #"VAL3", "None", #Type] |> &]

... But Not in 10.0.2

If we are running Mathematica version 10.0.1, this is the end of the story. But version 10.0.2 introduced a number of bugs into the type systems associated with queries (and fixed some too, to be fair). As a result, the queries listed above spew out some nonsensical messages. Such messages have appeared, and been discussed, in some other questions as well (e.g. 73299 and 73855).

In brief, the system can determine dataset types in two ways. For purposes of discussion, we will call the two ways static typing and dynamic typing. The first, static typing, inspects the actual data in a dataset and observes the types that are present. This is fairly reliable, although sometimes unusual heuristics present surprises. The second way, dynamic typing, is performed during query execution. It involves a customized evaluation procedure that inspects both the operations being performed and the data being operated upon. Unfortunately, the 10.0.2 implementation of dynamic typing exhibits a few evaluation leaks and these cause the kinds of errors discussed in these various questions.

10.0.2 Workaround?

The documented semantics of the query sublanguage are insensitive to the types of the data being operated upon. My guess is that this will change as that sublanguage develops, but today the type information is essentially unnecessary for query execution. Experimentation suggests that one can largely disable the dynamic type inferencing in 10.0.2 by defining a helper function like this:


ds_Dataset[noTypes[query__]] ^:= ds // Normal // Dataset`ShowPlan[Query@query] // Dataset

This function uses a crude approach that does the following:

  1. Extracts the data from the dataset using Normal.
  2. Generates a function that implements the query using Dataset`ShowPlan.
  3. Applies that function to the extracted data.
  4. Wraps the data back into a dataset, with the types being inferred by the fairly reliable static process.

Naturally, this modification voids all warranties. But I have found it useful when I am trying to determine whether a query doesn't work because I made an error, or because the type system made an error. The approach is clearly suboptimal, and might disable some lesser-used query functionality. But perhaps it is useful nonetheless.

In the case at hand, noTypes is used like this to produce the expected result without any warning messages:

ds @ noTypes[All, If[#"VAL1" == #"VAL2" == #"VAL3", <|#, "Type" -> "None"|>, #] &]


All of the assertions in this post concerning the operation of Query and Dataset are speculations arising from personal experimentation.

  • $\begingroup$ Thanks for the nice answer. I'm actually using 10.0.2, so I need the workaround. $\endgroup$
    – Lea
    Feb 17 '15 at 8:21
  • $\begingroup$ I observe now the trouble, that subsequent queries do not work anymore. Is there any way to reset the workaround? $\endgroup$
    – Lea
    Feb 19 '15 at 12:14
  • $\begingroup$ The workaround should only take effect if there is an explicit reference to noTypes. Do you have a counterexample? In any event, ClearAll[noTypes] will remove the definition. Also, restarting your session will clear it unless you have added to your init.m. $\endgroup$
    – WReach
    Feb 19 '15 at 12:39
  • $\begingroup$ Sorry for the late response, I'm only working part time. Maybe my trouble has nothing to do with your workaround, in my small example from the question, I can't reproduce it, the follwoing just works: onlyfeds = feds[Select[StringMatchQ[#"Type", "FE"] &]]. But in my more complex analysis, I get an error telling me: "bla blub is expected to have an Association as the first argument". Nevertheless, if I'm using the idea of m_goldberg it works. $\endgroup$
    – Lea
    Feb 20 '15 at 9:09
  • 1
    $\begingroup$ The solution given here (the first one) works in Mathematica 12, for anyone revisiting this solution. $\endgroup$
    – KBL
    Aug 1 '19 at 15:47

Although not elegant, I often find that working with a list of associations is easier than working directly in a dataset.

vals = Equal @@@ Normal[ds[All, Lookup[{"VAL1", "VAL2", "VAL3"}]]]
{True, True, True, False, True, True, False}
Dataset @ MapThread[
  Module[{assoc = #2}, If[#1, assoc["Type"], assoc["Type"] = "NONE"]; assoc] &, 
  {vals, ds // Normal}]


  • $\begingroup$ as a M beginner, this way wasn't very intuitive for me, but it work's well, even in version 10.0.2. Thanks! $\endgroup$
    – Lea
    Feb 19 '15 at 12:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.