Association's Key[key] extraction different in V10.1/V10.2

Question

V10.1:

<|a -> 1|>[Key @ a]
(* Missing["KeyAbsent", Key[a]]*)

V10.0:

<|a -> 1|>[Key @ a]
(* 1 *)

Bug or Design Change?

Update: This bug/design change remains in 10.2.

Even though my conclusions in a separate answer differ from the accepted @Szabolcs answer (not to mention the other answers and WRI's developers!) the approach of seeing the issue in a broader context is useful for clarifying the operation of any key extraction implementation.

Answer 1

Only the designer could say if this is a bug or a bugfix. But, like others, I also consider this change an improvement. What follows is just personal opinion on why, backed up by observations.

How would you design associations if your goal was to allow any expression to be a key (useful as shown by GroupBy, Counts, etc.) and also keep associations user friendly?

As I see, the primary function of Key is to disambiguate between numerical indices and keys that can be any expression in functions that take both.

We have two categories of functions. One that can take either numerical indices or keys, such as:

• Part, Extract, ReplacePart, etc.

And another one that takes only keys, such as

• <| ... |>[...], KeyExistQ, KeyTake, Lookup, etc.

Based on 10.1 behaviour, this appears to be the general behaviour:

• Functions that can take numerical arguments as well require key names to be wrapped by Key. An exception is strings, which are automatically treated as if they were already wrapped in Key. Any other expression is disallowed.

• Functions that can only take keys, interpret the key name literally.

These rules will make key handling convenient and unambiguous:

asc = <|a -> 1, 1 -> 2, Key[1] -> 3, "z" -> 4, Key["z"] -> 5,
    Key["w"] -> 6|>;

asc[[a]]

During evaluation of Part::pkspec1: The expression a cannot be used as a part specification. >>
(* <|a -> 1, 1 -> 2, Key[1] -> 3, "z" -> 4, Key["z"] -> 5, 
  Key["w"] -> 6|>[[a]] *)

asc[[Key[a]]]
(* 1 *)

asc[[1]]
(* 1 *)

asc[[Key[1]]]
(* 2 *)

asc[["z"]] (* automatically treated as Key["z"] *)
(* 4 *)

asc[[Key["z"]]]
(* 4 *)

asc[["w"]]
(* Missing["KeyAbsent", "w"] *)

asc[[Key@Key["z"]]]
(* 5 *)

asc[a]
(* 1 *)

asc[1]
(* 2 *)

asc[Key[1]]
(* 3 *)

asc["z"]
(* 4 *)

asc[Key["z"]]
(* 5 *)

asc[Key@Key["z"]]
(* Missing["KeyAbsent", Key[Key["z"]]] *)

Allowing both wrapped (Key[...]) and unwrapped names in situations where only key names are expected would lead to ambiguities. What would asc[Key["z"]] refer to? The key "z" or the key Key["z"]? As Mr. Wizard points it out, how could we even test reliably if a key exists or not?

You noticed that Lookup does indeed accept both wrapped and unwrapped keys, and this leads to some strange behaviour. Both Lookup[<|"z" -> 1|>, Key["z"], default] and Lookup[<|"z" -> 1|>, "z", default] give 1 even though Key["z"] =!= "z". This is what I'd consider a bug instead of wrapped forms not working where only names are expected.

You would be of course right if you said that using key names which themselves include Key is not a very good idea anyway, but if the language can be designed to avoid ambiguities, I think it is really better to do so.

Answer 2

Summary: Yes it is a "bug" and it also suggests changing [], #& and KeyTake's key extracting semantics:

Background: A Disambiguating Imperative Emerges:

While related to language design, subtlities such as these do impact de-bugging, the selection of apt data structures and internal consistency as a WL Data Science code bases mature. The issue here seems to be one of meta-language and disambiguation with a certain incoherency present with the current semantics.

The need for an initial disambiguation involved separating the specification of a part by (traditional) position and by (recently introduced) name. So while a construction like assoc[[1]] traditionally defines the first part of assoc, once keys are allowed to include arbitrary expressions, a potential conflict arises since assoc[[1]] can potentially also refer to the value of key, 1. The implemented disambiguation is to wrap Key over expressions intended to be interpreted as keys with no wrapping required for strings since their key status is implicit.

At first pass, assoc[Key[a]] doesn't seem subject to such a disambiguating imperative. The functional assoc[a] key extraction has no indexing precedent so arguments can apparently always be assumed to be literal keys. Hence, the usage assoc[Key[a]] appears redundant and presumably underpins the post V10.1 design decision to interpret it as the literal key, Key[a], (similar to all other literal key interpretations). Consequently, even with an Association containing the key, a, an error message is produced, with the "positive" affect of guiding users to the simpler usage assoc[a] (that is, without the "esoteric" Key[a] key already existing in the Association in which case its value would be returned instead).

A Design Oversight

I think this is a design oversight for the following reason:

First note (by the same reasoning) the following "redundancy":

{assoc[["a"]], assoc[[Key["a"]]]}

{1, 1}

That the second, assoc[[Key["a"]]], also returns 1 might seem like a polite nicety by way of ensuring users are not unduly penalized for displaying abundant caution, but it is actually more fundamental. One of the most powerful features IMO of the new Data Science functionality is the query operators and in particular, their composability (part of underpinning the development of operator forms) including the flexibility of its constituent operators. This means that, in general, a construct housing any key query knows not from whence it came.

So, for example, while the redundancy of assoc[[Key["a"]]] is clear enough, in general, another key, say Key[b], may well emerge from a prior collection of keys and code, in particular, as part of disambiguating an earlier [[Key[b]]] construction. If this key, as part of a chain of query compositions, subsequently appears as say assocA[Key[b]], its intended interpretation is clearly to, within assocA, find the value of the key, b. Hence this construct had better start indexing the key, b instead of heading off on a wild goose chase indexing the more "esoteric" key, Key[b] - which is what takes place in the current implementation (note that Query[Key[b]]@assocA works as expected and the link with the [] notation as part of Dataset queries).

This can be clarified by the following (where the "ambiguity concerns" raised by Mr. Wizard and Szabolcs in their answers are also addressed).

Unification through consistent Key-Descending Indexing

From this collection, it can be observed that Key[x]'s semantics differ depending on its housing construct: for [[]] and Lookup it means the key, x; conversely, for [] and KeyTake, it means the key, Key[x]. I'll label the former interpretation, descending-key indexing and the latter, literal-key indexing. Hence the current situation is as follows:

• [[ ]] is a key-descending, string-literal indexer
• Lookup is a key-descending, symbol-literal, string-literal indexer
• [], #&, KeyTake, are all key-literal, symbol-literal, string-literal indexers

What I claim is a much safer, more coherent design is instead:

• [[ ]] is a key-descending, string-literal indexer
• [], #&, KeyTake & Lookup, are all key-descending, symbol-literal, string-literal indexers

Hence the design change entails ensuring [], #& and KeyTake all become key-descending indexers.

In the previous collection shown in the image, this would mean resolving the current "bugs" to instead produce the following outputs

{assoc[Key[a]], #[Key[a]]&@assoc, KeyTake[assoc, Key[a]]}

{3, 3, <|a -> 3|>}

instead of the current {4, 4, <|a -> 4|>} (n.b, and to also become consistent with Query[Key[a]]@assoc)

Now let me adress Mr. Wizard and Szabolcs reasonable points (that also clarified my thinking) to show that in fact this design change is not ambiguous nor does it represent a loss of scope (unlike the current post-10.1 change which for the above reasons is ambiguous). While my sense is that the current key-literal indexing was originally motivated more by the apparent lack of a disambiguation imperative, they suggest that it may instead relate to another potential disambiguation imperative - that of distinguishing those cases where Key[a] is in fact a key (and hence the aforementioned "wild good chase" is nothing of the sort).

Firstly, I think this is unlikely to be the case as it doesn't seem to me to be a natural thing to want to do - include Key[a]->1 as part of an Association? If for whatever reason it is implemented however, then consistent, key-descending indexing handles it simply and unambiguously - just use Key[Key[a]]. Sure, to those crazy Key[a]->1 key constructors, this would seem more unwieldy but equally, it would be bizarre to accommodate a weird construction at the expense of the entire coherency of WL's key-extracting mechanism (further, if Key[a]->1 really was a widespread desired element of an Association then it could always be considered as a->1 for key extraction purposes).

Coherency, Pedagogy and Backward Compatibility Advantages

Finally, any design change needs to incorporate backward compatability but the nice thing about this change is that the main effect would simply be to ensure that previous error messages no longer appear from users either applying "abundant caution" or else following the same pattern/expectations inculcated from previous descending-key indexing. The only change in terms of non-error-message output would be the esoteric usage just described which I suspect would be non-existent at any rate).

Most important of all however, this change would mean that potential inconsistencies in future Key[a] interpretations can no longer occur something that IMO has only not yet bitten due to both shy users and WL's still maturing process in weighty data science applications.

Answer 3

I am going to merely guess that it is an intentional change as:

1. there exist other ways to extract the same element, e.g.:

   <|a -> 1|>[[Key[a]]]
   
   Key[a] @ <|a -> 1|>
   

2. it is more general to allow verbatim Key within keys

   KeyExistsQ[<|a -> 1|>, Key[a]]        (* false *)
   
   KeyExistsQ[<|Key[a] -> 1|>, Key[a]]   (* true *)
   

Answer 4

In V10.1

a = 42;
<|a -> 1|>[a]

gives

1

I look at this as a fix for a bug or design flaw that made the use of symbol keys awkward in V10.0. The loss of the <|a -> 1|>[Key @ a] is a small price to pay for what we have gained.