# Lookup with default broken in Dataset

Given an Association,

data = <|"1" -> <|"A" -> 10, "B" -> 20|>,
"2" -> <|"C" -> 30, "B" -> 40|>|>


Lookup works w/ default:

 data // Map[Lookup[#, "A", 0] &]


<|"1" -> 10, "2" -> 0|>

But not in Dataset (output is the FullForm)

data // Dataset // Query[All, Lookup[#, "A", 0] &]


The problem occurs because the three-argument form of Lookup is not recognized by the type inferencing performed by Dataset. One work-around is to use Query directly, bypassing the type inferencing:

data // Query[All, Lookup[#, "A", 0]&]
(* <| "1" -> 10, "2" -> 0 |> *)


Another (nasty) work-around is to "hide" the symbol Lookup from the type inferencer by expressing it instead as Lookup&[]:

data // Dataset // Query[All, Lookup&[][#, "A", 0]&]


Discussion (current as of version 10.1)

A Dataset query starts by attempting to infer the types of its arguments for various purposes. The responsible function is TypeSystemTypeApply. Notionally, this function can infer the type of any valid Wolfram expression. I say "notionally" because I am not sure whether it is intended to be able to handle all expressions, or only a query-friendly subset.

In any event, TypeApply has special handling for certain operators. We can see at least some of those operators by evaluating this expression:

{TypeSystem$Signatures, TypeSystem$SubSignatures} // Query[Flatten /* Sort, Keys]

(* {AllTrue,AnyTrue,Append,...,Lookup,...,Values,Variance,VectorQ,Vectors} *)


Lookup is one of the special operators. We can see which forms of Lookup are accepted by evaluating:

printSignatures[s_Symbol] :=
Block[{$ContextPath = {"TypeSystem","TypeSystemZSignaturesPackagePrivate"} ~Join~$ContextPath }
, Print @ Column @ TypeSystem\$Signatures[s][[All, 1]]
]

printSignatures[Lookup]
(*
{s_Struct, keys:{__String}}
{Vector[Assoc[k_,v_,_],m_], Vector[k_,n_]}
{Assoc[k_,v_,_], Vector[k_,n_]}
{Assoc[k_,v_,_], k_}
{Vector[Assoc[k_,v_,_],m_], k_}
{s_Struct, key_String}
{Vector[s_Struct,n_], key_}
*)


We can see that all of these signatures accept only two arguments. The three-argument form is not accepted. This is the origin of the error message that we get:

Expression of the form Lookup[Association[(_String -> _Integer)..], "A", 0] cannot be evaluated.

The phrase cannot be evaluated in this case actually means cannot be have its type inferred by TypeApply.

All of this talk of signatures is only relevant for the recognized operators. Unrecognized operators receive only light treatment, and often end up with a generic type annotation.

The exhibited expression works in versions 10.0.1, but fails in versions 10.0.2 and 10.1. The reason is that starting with 10.0.2, TypeApply started looking into Function expressions. Prior to 10.0.2, the Lookup[...]& expression would have been treated as an unrecognized function and would be passed through without signature analysis. But now that Function is transparent, the signature restriction is causing difficulties.

We can see how the nasty work-around expression Lookup&[] works: it is treated by TypeApply as an unknown operator. TypeApply does not (presently) evaluate Function expression heads before performing signature analysis. This could change at any time.

It is not clear to me whether this restriction of Lookup to two arguments is intentional or an oversight. The inferred type of three-argument Lookup applied to an Association would need to be the least supertype containing the third argument type and the association value type.

Almost none of this discussion comes from official WRI documentation. Take it with a grain of salt.

• Thanks for writing these detailed reports. – alancalvitti Jun 7 '15 at 20:05
• Sometimes your first solution data // Query[All, Lookup[#, "A", 0]&] doesn't work, whereas Lookup&[] does when used directly data[All, Lookup &[][#, keys, 0] &] or as a separate Query. – alancalvitti Jun 17 '15 at 18:20
• @alancalvitti I'd be interested to know about a case where the first solution fails for a non-dataset object since that would be an entirely different class of problem. If the details are lengthy, it might be best to discuss it in chat rather than the comments here. – WReach Jun 17 '15 at 18:51
• Let me clarify, I meant in the context of Dataset. Nevertheless, 1st case doesn't work while the 2nd does - and thanks for the &[] syntax - new to me. – alancalvitti Jun 18 '15 at 18:27
• @alancalvitti @WReach This is fixed in 10.2. You can now write TypeSystemSignatures[Lookup] and you'll see a nice table of what the rules are. – Taliesin Beynon Jul 2 '15 at 7:51

This isn't really an answer, but it can't be written as comment.

I'm running V10.1 on OS X. I get the same results, so I also think it's a bug. I note the problem seems to be the presence of the 3rd argument, even if it not used.

data = <|"1" -> <|"A" -> 10, "B" -> 20|>, "2" -> <|"C" -> 30, "B" -> 40|>|>;
ds = Dataset[data]


ds[All, Lookup[#, "B"] &]


ds[All, Lookup[#, "B", 0] &]
`