The problem with the query proposed in the question is that it is attempting to apply Select
to each row. As @Kuba points out in a comment, the use of Select
is unnecessary. The query can be expressed like this:
ds[All, If[#"VAL1" == #"VAL2" == #"VAL3", <|#, "Type" -> "None"|>, #] &]
The same result can be obtained by an alternate formulation:
ds[All, <| #, "Type" -> If[#"VAL1" == #"VAL2" == #"VAL3", "None", #Type] |> &]
... But Not in 10.0.2
If we are running Mathematica version 10.0.1, this is the end of the story. But version 10.0.2 introduced a number of bugs into the type systems associated with queries (and fixed some too, to be fair). As a result, the queries listed above spew out some nonsensical messages. Such messages have appeared, and been discussed, in some other questions as well (e.g. 73299 and 73855).
In brief, the system can determine dataset types in two ways. For purposes of discussion, we will call the two ways static typing and dynamic typing. The first, static typing, inspects the actual data in a dataset and observes the types that are present. This is fairly reliable, although sometimes unusual heuristics present surprises. The second way, dynamic typing, is performed during query execution. It involves a customized evaluation procedure that inspects both the operations being performed and the data being operated upon. Unfortunately, the 10.0.2 implementation of dynamic typing exhibits a few evaluation leaks and these cause the kinds of errors discussed in these various questions.
10.0.2 Workaround?
The documented semantics of the query sublanguage are insensitive to the types of the data being operated upon. My guess is that this will change as that sublanguage develops, but today the type information is essentially unnecessary for query execution. Experimentation suggests that one can largely disable the dynamic type inferencing in 10.0.2 by defining a helper function like this:
Needs["Dataset`"]
ds_Dataset[noTypes[query__]] ^:= ds // Normal // Dataset`ShowPlan[Query@query] // Dataset
This function uses a crude approach that does the following:
- Extracts the data from the dataset using
Normal
.
- Generates a function that implements the query using
Dataset`ShowPlan
.
- Applies that function to the extracted data.
- Wraps the data back into a dataset, with the types being inferred by the fairly reliable static process.
Naturally, this modification voids all warranties. But I have found it useful when I am trying to determine whether a query doesn't work because I made an error, or because the type system made an error. The approach is clearly suboptimal, and might disable some lesser-used query functionality. But perhaps it is useful nonetheless.
In the case at hand, noTypes
is used like this to produce the expected result without any warning messages:
ds @ noTypes[All, If[#"VAL1" == #"VAL2" == #"VAL3", <|#, "Type" -> "None"|>, #] &]
Disclaimer
All of the assertions in this post concerning the operation of Query
and Dataset
are speculations arising from personal experimentation.
ds[All, If[#"VAL1" == #"VAL2" == #"VAL3", <|#, "Type" -> "None"|>, #] &]
this works but gives errors. I don't know why $\endgroup$