Finding out all undocumented values for "Properties" in XxxData[] functions

Question

No, XxxData[] is not a new function in Mathematica 11. I am just using that name for all curated data-related functions.

As you know, CountryData[] is one of such so called "curated data" functions in Mathematica. It basically presents a convenient way to access lots of information on all countries of the today's world.

In this question, an undocumented value for property passed to CountryData[] is discussed ("OfficialLanguages").

How to find out all undocumented values?

The same question for:

AstronomicalData[]
GraphData[]
WordData[]
...

(all XxxData[] from Mathematica's curated data)

I tried examining correspondent mx file, but no results. Is there a smart way?

---

I read several related questions here, that deal with undocumented options and other parameter values for certain Mathematica's functions, but none of presented methods for discovering undocumented values worked for CountryData[].

Answer 1

The standard way to retrieve the list of properties available for most of the curated data functions is to ask for the "Properties" property, e.g.

CountryData["Properties"]
(*
  {"AdultPopulation", "AgriculturalProducts", <<219>>, "WaterArea", "WaterwayLength"}
*)

or

AstronomicalData["Properties"]
(*
  {AbsoluteMagnitude,AbsoluteMagnitudeH,<94>>,TholenSpectralType,VisualBandMagnitude}
*)

As the question observes, this list of properties is not always complete. The reason is that the property list is actually computed from the various data sources that provide the data. It is not always feasible to compute a comprehensive list.

CountryData

As an example, consider CountryData. If we ask for a list of country properties, we find that there are 223:

CountryData["Properties"] // Length
(* 223 *)

"OfficialLanguages" is not on that list.

MemberQ[CountryData["Properties"], "OfficialLanguages"]
(* False *)

Yet we can ask for it:

CountryData["Tuvalu", "OfficialLanguages"]
(* {Entity["Language", "Tuvaluan"], Entity["Language", "English"]} *)

The clue lies in the fact that, as of version 10, CountryData returns entities:

CountryData["Tuvalu"]
(* Entity["Country", "Tuvalu"] *)

If we ask for the properties of a country entity, we find that the list is much larger:

CountryData["Tuvalu"] // EntityProperties // Length
(* 747 *)

"OfficialLanguages" is on that list:

CountryData["Tuvalu"] // EntityProperties //
  MemberQ[EntityProperty["Country", "OfficialLanguages"]]
(* True *)

So, as it turns out, when CountryData is asked about a property it first consults the list of built-in properties. If that fails, then the entity framework is consulted.

AstronomicalData

By contrast, AstronomicalData is not yet using the entity framework:

AstronomicalData["Mars"]
(* "Mars" *)

Thus, the built-in properties are simply the ones returned by AstronomicalData["Properties"].

Or are they?

The General Case

Often, we can find out where data properties are coming from by inspecting the definitions of the functions themselves. For example, we can look at the definitions for CountryData by issuing the statement:

Debug`$ExamineCode = True;

??CountryData

A close inspection reveals the fallback behaviour to the entity framework discussed above, and also that the built-in properties are associated with a symbol called DataPaclets`CountryDataDump`$PropertyHash:

?? DataPaclets`CountryDataDump`$PropertyHash

(*
   DataPaclets`CountryDataDump`$PropertyHash[{AdultPopulation}]=1

   DataPaclets`CountryDataDump`$PropertyHash[{AgriculturalProducts}]=2

   DataPaclets`CountryDataDump`$PropertyHash[{AgriculturalValueAdded}]=3

   ...

   DataPaclets`CountryDataDump`$PropertyHash[{{InflationRate,Local},Date}]=419
*)

Observe that there are 419 properties in that list, even though CountryData["Properties"] only listed 223. This is a consequence of the fact that the property list is actually computed from a larger data source.

Of course, any information gained by this means is undocumented, unsupported, and liable to change at any time. In the case of paclet data, "any time" could even mean between sessions -- not just between releases.

This analysis was conducted using Mathematica version 10.0.1.

Answer 2

What happens is that if a requested property p is not one of the standard "Properties", CountryData then computes

EntityValue[Entity["Country", <standard name>], p]

For example, the possible entity properties for a country are found by

EntityProperties["Country"]
(*    
{EntityProperty[Country, AdjustedNetNationalIncome],
 EntityProperty[Country, AdjustmentSeasonalBorrowings],
 EntityProperty[Country, AdministrativeDivisions],
 EntityProperty[Country, AdultPopulation],
 <<740>>,
 EntityProperty[Country, Workforce],
 EntityProperty[Country, WPI]}
*)

"Official Languages" is in this list.

AstronomicalData does not follow the same procedure. For example

AstronomicalData["Mars", "ApoapsisTimeLast"]               (* --> AstronomicalData::notprop... *)
EntityValue[Entity["Planet", "Mars"], "ApoapsisTimeLast"]  (* --> Thu 2 Jan 2014 *)

Apparently each data area is curated in its own way, but perhaps some follow the CountryData model. Here my free time runs out. :)