3
$\begingroup$

Update: I solved the problem: the error will occur when the length of an association is less than the length of the longest association in the association of associations


In my original problem I said:

I can't see the difference between the data that might be causing the error. My only thought is that variables that create the error have fewer years of data than the variables that do not create the error (I haven't comprehensively tested this).

Note that in my database it does not create "year"->"KeyAbsent" for years that are outside the range of years for the variable in question, but within the maximum range of years for all variables. i.e. if var1 covers 2000 to 2005 but var2 covers 1980 to 2010, var1 is not padded in the database with "1980"->"KeyAbsent","1981"->"KeyAbsent",etc.. This does occur when I try and create a toy database to debug things.

Turns out this is exactly the problem. When I standardise the number of years the error does not occur. This is a little inelegant as I Merge the data with a list of "year"->Missing[] that spans the full range of years, so the dataset is much larger and mostly contains Missing[] values, but such is life.

I guess this shows that Dataset needs very well-behaved data.

My question now is: should this be reported as a bug?

Original problem

Dataset is failing to present data in some instances but not others when I insert new data.

I will first describe the database, and then describe the problem, before ending with the questions I have. For reference I am using Mathematica 10.2.

The database

The dataset is an association of variable names, and each variable name is the key for an association of data relevant to the variable. This includes a list of metadata, such as units, data source etc. and a key "data", which is an association of countries, where each country is the key for a further association of years and data for that year, i.e. data is a matrix of countries and years, represented as an association of associations.

To work backwards, this association of associations is associated with a variable, which is itself in an association, all wrapped up as a dataset.

I therefore have a database that contains a set of variables for which I have a panel of data and relevant metadata. I'd post all the code but:

  • the code is quite long
  • a lot of the data isn't public domain
  • my problem isn't with the database itself (I think), just some strange behaviour when I add new data

Below is an example of the data structure using some random data:

<|"var1" -> <|"metadata1" -> "foo", "metadata2" -> "bar", "data" -> <|"country1" -> <|"2013" -> 9.778361595825281*^6, "2014" -> 6.109805514945384*^6, "2015" -> 8.075575892851099*^6|>, "country2" -> <|"2013" -> 8.170001132219501*^6, "2014" -> 6.592762837990681*^6, "2015" -> 3.2318544857920036*^6|>|>|>, "var2" -> <|"metadata1" -> "foo", "metadata2" -> "bar", "data" -> <|"country1" -> <|"2013" -> 1.0822109010023326*^6, "2014" -> 3.3734311193888187*^6, "2015" -> 9.272459105590582*^6|>, "country2" -> <|"2013" -> 9.399156851764016*^6, "2014" -> 2.4352994845302533*^6, "2015" -> 9.28416944498226*^6|>|>|>, "var3" -> <|"metadata1" -> "foo", "metadata2" -> "bar", "data" -> <|"country1" -> <|"2013" -> 978119.4832605273, "2014" -> 2.408638112675734*^6, "2015" -> 3.905368882210683*^6|>, "country2" -> <|"2013" -> 6.690702513293147*^6, "2014" -> 1.6114252243915442*^6, "2015" -> 6.125816084110443*^6|>|>|>|>

The problem

I then need to add new countries to a particular variable and save this to the database. I do this using insert:

exampledataset = Insert[exampledataset, newcountries, {"var1", "data", -1}];

I have tried every way of adding data I can find (Join, Append, Catenate, Merge), both using Query and transforming the database to an association using Normal and then adding the data. All produce the same problem below.

(BTW Insert is the easiest, simplest way of adding data to a Dataset that returns the whole database, while Append is best if you just want to return the part you of the database you have appended data to.)

Inserting the new data and saving the database works fine. The problem I get is that for only some variables, when I call them (either an added country or a country originally in the database), I get the following error in the Notebook:

error message

And the following error in Messages:

A GridBox with an invalid first argument was encountered. The first argument to GridBox must be a rectangular matrix (nested list).

(In the example above I am calling the GDP per capita (in PPP terms) for the UK (GBR), which was a country originally in the database, the same error occurs for a country inserted into the database).

The data is still there, and I can for instance use it in DateListPlot or use Normal to see the data, it just won't be presented as a Dataset.

What is very strange is that this error does not appear for other variables. To illustrate, below is the FullForm code for two variables. The first causes an error (and if you copy this into Mathematica you should be able to recreate the error). The second returns a properly formatted Dataset table.

Variable that produces error

Dataset[<|"1990" -> 17984.947579055202, "1991" -> 18282.159065384836, "1992" -> 18640.849891426107, "1993" -> 19513.13163323808, "1994" -> 20506.754375429962, "1995" -> 21072.568621120452, "1996" -> 22408.69696686949, "1997" -> 23720.41048510612, "1998" -> 24458.574731843535, "1999" -> 25149.53530077939, "2000" -> 27339.99283903917, "2001" -> 28686.09596998566, "2002" -> 29981.094440667635, "2003" -> 31189.310758855536, "2004" -> 33093.51202785343, "2005" -> 34525.40579542566, "2006" -> 36832.94040547561, "2007" -> 37423.18191685278, "2008" -> 37752.95726175876, "2009" -> 36285.80796949102, "2010" -> 35920.02193529807, "2011" -> 36549.422798983716, "2012" -> 37385.9234744073, "2013" -> 38255.15335756281, "2014" -> 39136.77423643249|>, TypeSystem`Struct[{"1971", "1972", "1973", "1974", "1975", "1976", "1977", "1978", "1979", "1980", "1981", "1982", "1983", "1984", "1985", "1986", "1987", "1988", "1989", "1990", "1991", "1992", "1993", "1994", "1995", "1996", "1997", "1998", "1999", "2000", "2001", "2002", "2003", "2004", "2005", "2006", "2007", "2008", "2009", "2010", "2011", "2012", "2013"}, {TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real]}], <|"Origin" -> HoldComplete[Query["GDPcapitaPPP", "data", "GBR"][Dataset`DatasetHandle[25048302199938]]], "ID" -> 41639760864439|>]

Variable that does not produce error

Dataset[<|"1971" -> 8.736864*^6, "1972" -> 8.801008*^6, "1973" -> 9.130209*^6, "1974" -> 8.792701*^6, "1975" -> 8.347073*^6, "1976" -> 8.512489*^6, "1977" -> 8.694163*^6, "1978" -> 8.647694*^6, "1979" -> 9.087685*^6, "1980" -> 8.307931*^6, "1981" -> 8.055457*^6, "1982" -> 8.009529*^6, "1983" -> 7.966936*^6, "1984" -> 7.962121*^6, "1985" -> 8.407001*^6, "1986" -> 8.538415*^6, "1987" -> 8.606847*^6, "1988" -> 8.683017*^6, "1989" -> 8.66572*^6, "1990" -> 8.621494*^6, "1991" -> 8.914765*^6, "1992" -> 8.88217*^6, "1993" -> 8.97176*^6, "1994" -> 9.041684*^6, "1995" -> 9.056854*^6, "1996" -> 9.445435*^6, "1997" -> 9.178926*^6, "1998" -> 9.273166*^6, "1999" -> 9.296416*^6, "2000" -> 9.334714*^6, "2001" -> 9.36858*^6, "2002" -> 9.139805*^6, "2003" -> 9.297911*^6, "2004" -> 9.276774*^6, "2005" -> 9.321445*^6, "2006" -> 9.167183*^6, "2007" -> 8.835245*^6, "2008" -> 8.670822*^6, "2009" -> 8.181183*^6, "2010" -> 8.43323*^6, "2011" -> 7.826494*^6, "2012" -> 8.04831*^6, "2013" -> 7.964277*^6|>, TypeSystem`Struct[{"1971", "1972", "1973", "1974", "1975", "1976", "1977", "1978", "1979", "1980", "1981", "1982", "1983", "1984", "1985", "1986", "1987", "1988", "1989", "1990", "1991", "1992", "1993", "1994", "1995", "1996", "1997", "1998", "1999", "2000", "2001", "2002", "2003", "2004", "2005", "2006", "2007", "2008", "2009", "2010", "2011", "2012", "2013"}, {TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real], TypeSystem`Atom[Real]}], <|"Origin" -> HoldComplete[Query["TPES", "data", "GBR"][Dataset`DatasetHandle[119769510946048]]], "ID" -> 35055575999750|>]

I can't see the difference between the data that might be causing the error. My only thought is that variables that create the error have fewer years of data than the variables that do not create the error (I haven't comprehensively tested this).

Note that in my database it does not create "year"->"KeyAbsent" for years that are outside the range of years for the variable in question, but within the maximum range of years for all variables. i.e. if var1 covers 2000 to 2005 but var2 covers 1980 to 2010, var1 is not padded in the database with "1980"->"KeyAbsent","1981"->"KeyAbsent",etc.. This does occur when I try and create a toy database to debug things.

What is also weird is that the act of inserting the data changes the TypeSystem (for both error-creating and error-free variables) as shown below:

FullForm of variable before inserting new data

Dataset[<|"1990" -> 17984.947579055202, "1991" -> 18282.159065384836, "1992" -> 18640.849891426107, "1993" -> 19513.13163323808, "1994" -> 20506.754375429962, "1995" -> 21072.568621120452, "1996" -> 22408.69696686949, "1997" -> 23720.41048510612, "1998" -> 24458.574731843535, "1999" -> 25149.53530077939, "2000" -> 27339.99283903917, "2001" -> 28686.09596998566, "2002" -> 29981.094440667635, "2003" -> 31189.310758855536, "2004" -> 33093.51202785343, "2005" -> 34525.40579542566, "2006" -> 36832.94040547561, "2007" -> 37423.18191685278, "2008" -> 37752.95726175876, "2009" -> 36285.80796949102, "2010" -> 35920.02193529807, "2011" -> 36549.422798983716, "2012" -> 37385.9234744073, "2013" -> 38255.15335756281, "2014" -> 39136.77423643249|>, TypeSystem`Assoc[TypeSystem`Atom[String], TypeSystem`Atom[Real], TypeSystem`AnyLength], <|"Origin" -> HoldComplete[Query["GDPcapitaPPP", "data", "GBR"][Dataset`DatasetHandle[27711181911732]]], "ID" -> 75728916293122|>]

Questions

So my questions are:

  1. Should I worry about this error occurring? i.e. does it matter for the integrity of the database? Or is it just a display problem?

If I should worry about this error:

  1. Why is the error occurring?
  2. How can I prevent it from occurring?

Any help would be much appreciated. Dataset has made most things much easier but not being able to add new entries without errors is a limitation.

Thanks.

$\endgroup$
  • 1
    $\begingroup$ I think it's a bug in the typesetting code, it seems like a perfectly valid Dataset -- things only break when trying to display it. $\endgroup$ – ilian Sep 5 '15 at 16:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.