Query/Part Bug with :>

Question

Bug observed in 10.0.1 and persisting through to 10.4.0

Part can fail with :>

 d = <|"a" :> <|"b" -> <|"c" -> 1, "d" -> 2|> |>|>;

 d[["a", "b", All]]

 (* 
    Part::pspec1: Part specification b is not applicable. >>

    Missing["KeyAbsent", "a"]

  *)

something Query seems immune to ...

     d // Query["a", "b", All]

  (* <|"c" -> 1, "d" -> 2|>  *)

... well almost

 {"a" :> 1} // Query["a"]

 (* Missing["PartInvalid", "a"] *)

Confirmed Bugs?

Answer 1

Looking at this again I think it is not a bug but a design limitation. Despite the FullForm appearing similar an Association is not actually an atomic Association object until it is evaluated. Take for example:

AtomQ[<|a -> 1, b -> 2, c -> 3, d -> 4|>]    (* True *)

AtomQ[Unevaluated[<|a -> 1, b -> 2|>]]       (* False *)

One may substitute AssociationQ for AtomQ with the same result.

Note that Unevaluated does not prevent an Integer from being recognized as atomic:

AtomQ[Unevaluated[2]]  (* True *)

Given this if use of :> prevents evaluation, as I believe it does and should, the expression <|"b" -> <|"c" -> 1, "d" -> 2|> |> in your example is not an actual Association object until it is evaluated, therefore e.g. Part (which is simpler by far than Query) sees no deeper Keys to extract. However if we let it pull the Value of "a", which causes it to evaluate, we can extract the Key "b" without issue:

d = <|"a" :> <|"b" -> <|"c" -> 1, "d" -> 2|>|>|>;

d[["a"]][["b"]]

<|"c" -> 1, "d" -> 2|>

d[["a"]][["b", "c"]]

1

I feel the issue is similar to this:

Cases["a" -> 1/2, _Rational]
Cases["b" :> 1/2, _Rational]

{1/2}

{}

Cases["a" -> 3 I, _Complex]
Cases["b" :> 3 I, _Complex]

{3 I}

{}

i.e. 1/2 and 3 I are not Rational or Complex until released for evaluation.

---

Regarding the final example {"a" :> 1} // Query["a"] I think that may be considered a bug as if Query works with simple Rule lists I think it should work here too.

Answer 2

Yes, it seems to be a bug firstly because different output comes from different ways of specifying a Part

d = <|"a" -> <|"b" :> <|"c" -> <|"d" :> "e"|>|>|>|>;

d[["a", "b", All]]    (* <|"c" -> <|"d" :> "e"|>|> *)
d[["a", "b", 1]]      (* "c" -> <|"d" :> "e"|> *)
d[["a", "b", "c"]]    (* Missing["KeyAbsent", "b"]  *)

Mr Wizard's answer speculated that the cause of this might be due to when/how an Association is interpreted (or loses it atomicity) and while the above example tends to suggest otherwise, his reasoning might go some way to explaining the bug's origin. Certainly it appears as if some initial "whole processing" is taking place as (opposed the sequential d[["a"]][["b"]]) as the following analysis shows (and without being privy to the no-doubt many (competing) imperatives of Cases, Part, Query etc). Firstly, it seems to me that the example,

Cases["a" :> 3 I, _Complex]  (* {} *) 

(* anomalously IMO - {3 I} is more natural *)

should instead yield {3 I} (as should also Cases["a":>3 I, 3 I]) since it's more analogous to the second expression of the following as opposed to the fourth (since the number's Complex nature is a parsing issue more than an evaluative one).

Cases["b" -> 1, _Integer]   (* {1} *)
Cases["b" :> 1, _Integer]   (* {1} *)
t = 1;
Cases["b" -> 1, _Integer]   (* {1} *)
Cases["b" :> t, _Integer]   (*  {}  *)

Compare the last evaluation with

Cases["a" :> 3 I, _?NumberQ]    (* {3 I}  *)
Cases["b" :> t, _?IntegerQ]     (* {1}  *)

which are both expected since now each part gets evaluated in the test. Hence Cases respects the evaluation instruction in ->, :> while looking at an expression's structure only overiding this when explicitly asked to do so (such as in a test). Similarly, Part respects ->, :> evaluation instruction when looking at an Association's structure

t = <|"b" -> 1|>;

D1 = <|"a" -> t|>;
D2 = <|"a" :> t|>;

D1[["a", "b"]]   (* 1 *)
D2[["a", "b"]]   (* Missing["KeyAbsent","a"] *)

Query however, will overide :>'s delayed-evaluation to avoid not finding anything

D1 // Query["a", "b"]  (* 1 *) 
D2 // Query["a", "b"]  (* 1 *)

Sometimes this can lead to apparently anomalous, but probably, ultimately useful behaviour

s[2] = <|"b" -> 1, "c" -> 3|>;

D3 = <|"a" -> s[2]|>;
D4 = <|"a" :> s[2]|>;

D3[["a", 1]]   (* 1 *)
D3[["a", 2]]   (* 3 *)

D4[["a", 1]]   (* 2 *)
D4[["a", 2]]   (* Missing["KeyAbsent","a"] *)
D4 // Query["a", 1]   (* 2 *)  (* not 1 as might be expected as per below  *)
D4 // Query["a", 2]   (* 3 *)  (* apparently anomalous but probably useful *)

When it comes to more deeply nested associations, Part allows this :> evaluative instruction to (IMO anomalously) obscure the initial structure provided

D5 = <|"a" -> <|"b" -> 1|>|>;
D6 = <|"a" :> <|"b" -> 1|>|>;
D5[["a", "b"]]   (* 1 *)
D6[["a", "b"]]   (* Missing["KeyAbsent","a"] *) (* anomolously IMO - 1 is more natural *)

unlike Query which as per its usual earnest self invariably pulls out all stops (or at least parts)

D5 // Query["a", "b"]  (* 1 *)
D6 // Query["a", "b"]  (* 1 *)

In other words, to remove these claimed anomalies with deeply nested Associations, two Associations that are identical modulo Rule, RuleDelayed (both in unevaluated an evaluated forms) need to have the same output when Part operates on them - an eventuality guaranteed with Unit Tests satisfing this condition doubling up with Rules replaced with RuleDelayeds.

N.B. The other observed behaviour/formatting anomalies seem more clearcut

{"a" -> 1} // Query["a"]  (* 1 *)
{"a" :> 1} // Query["a"]  (* Missing["PartInvalid", "a"] *)

<|"a" -> <|"b" -> 1|>|>    (* <|"a" -> <|"b" -> 1|>|> *)
<|"a" :> <|"b" -> 1|>|>    (* <|"a" :> Association["b" -> 1]|> *)