Hot answers tagged

77

Here are a few ways, each of which operates upon the individual component associations. In the following discussion, recall that when a key name is not a valid symbol we can write, for example, #["col_name"] instead of #col. We can explicitly construct a new association that includes all of the old columns and adds a new one: ds[All, <| "col1"->"...


49

A Dataset represents an abstraction over a structured collection of data. Notionally, it is restricted to "well-behaved" data -- data that comes in simple forms that can be readily interchanged with external systems such as relational databases, XML documents, JSON documents, etc. These are commonplace forms such as vectors, records ("structs"), tuples, ...


38

Theoretically, Dataset supports any number of columns. The behavior you are seeing is actually because the type deduction that Dataset is doing behind the scenes isn't perfect (and indeed in some sense cannot be perfect). Your synthetic example is such that your second list of associations is "most consistent" with a particular type that doesn't typeset as ...


37

This response defines a function called traceTypes which provides a quick-and-dirty visualization of type system operation. The function is somewhat fragile as it depends upon undocumented implementation details in version 10.2. Despite this fragility, it might be useful for study purposes as it handles many common type system use cases. The code for the ...


37

Official Statement In case some of you missed it: There was an official reply from one of the WRI devs recently in our chat Hi. This is Jose, from Wolfram. We are aware of an unacceptable slowdown in some Dataset expressions, due to a bad dynamic interaction with the summary boxes of some objects. Both the TimeSeries objects of the coronavirus datasets ...


34

I'm the developer of Dataset. Yes, this is a gross documentation oversight. We planned this functionality but had to push it back to a point release. Somehow no-one caught this piece of legacy documentation. I've filed a bug on the documentation problem right now, it's easy to fix. As for when L-value assignment will be available, I'm hoping 10.0.1 or 10....


30

If you are willing to use an undocumented and future-incompatible hack you can set the global variable Dataset`FormattingCompound`PackagePrivate`$Limit to larger than its default value of 64 to see more of the dataset. For example, Dataset[RandomInteger[10, {1000, 5}]] shows 16 rows by default, but by doubling $Limit to 128 you will see 32 rows, etc. I will ...


28

Another way that works (for one or more columns) is: ds[All, <|#, "col3" -> #col1 + #col2, "col4" -> #col1 - #col2|> &] This gives: Also, sometimes the values for the new column might not be straightforwardly computed row by row. For example, you might have calculations like this: newcol = RotateLeft @ Normal[ds[All, (#col1 + #col2 &)...


27

Until scrollbars are implemented... ... I have found this little function for simple tabular data to work out fine with everyday work. One can size the viewer window (and probably use the tricks posted so far to extend the range of formatted output?). Also you can set a specific column (numerically) to be a repeated column, eg. to remain fixed while the ...


25

data = N @ Normalize[#, Total] & @ Counts @ Characters @ ExampleData[ {"Text", "DeclarationOfIndependence"} ]; Dataset @ data Dataset[KeyValueMap[<|"char" -> #, "freq" -> #2|> &, data]]


24

Inspired by WReach's answer, I started playing with Query based approach and here's what I came up with: data = {"days" -> {1, 2, 6, 8}, "area" -> {3, 6, 8, 2}, "frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}} With the data in the above form, we just create a Dataset simply as follows: dataset = Dataset[data]; Don't worry ...


23

This is a corner case that occurs when ascending operators are interleaved between descending operators. This case falls into an undocumented grey area. In the Details and Options section of the Query documentation, we read the following special rule: When one or more descending operators are composed with one or more ascending operators (e.g. desc /* ...


23

One simple way of "filtering" your data is to treat the points as a graph, and search for the shortest path from left to right: xScale = 10.; xy = Transpose[{N[Range[Length[data]]]*(xScale/Length[data]), data}]; start = {-xScale, Mean[data]}; finish = {2*xScale, Mean[data]}; graph = NearestNeighborGraph[Join[{start}, xy, {finish}], 25]; graph = ...


23

Dataset was restructured in the 12.1 release in order to support expanded formatting options and interactivity such as hiding and sorting. As a result, some Dataset outputs showed a slowdown due to inefficiencies in the dynamic output structures they produce. Because the code of Dataset is automatically field upgradable, we have released an update to its ...


19

I was playing around, trying to come up with a purely query-based solution. The result is by no means natural, but perhaps it holds some academic interest or will spark some insight into a better answer: Dataset[ { "days" -> {1, 2, 6, 8} , "area" -> {3, 6, 8, 2} , "frequency" -> {1, 4, 4, 2} , "height" -> {2, 3, 11, 6} } ][ Transpose, ...


19

We can explicitly construct a new association with key names of our choosing: dataset[All, <| "a" -> "a", "h" -> "b", "c" -> "c" |>] Alternatively, a function could be applied to the keys: dataset[All, KeyMap[# /. "b" -> "h" &, #] &] Note that a bug in the V10.0.0 type system prevents us from using the operator form KeyMap[# /. "...


19

We can start by importing the file as an XMLObject: $url = "https://dl.dropboxusercontent.com/u/1012958/iTunes%20Library.xml"; $xml = Import[$url, {"XML", "XMLObject"}]; Short[$xml, 4] (* XMLObject[Document][ { XMLObject[Declaration][Version->1.0,Encoding->UTF-8] , XMLObject[Doctype][plist,Public->-//Apple Computer//DTD PLIST 1.0//EN,&...


17

In lieu of Set, the Query syntax offers various ways to update selective elements of a dataset. For example, we can change the value of the field a in the first row like this: ds[{1 -> (<| #, "a" -> 999|> &)}] or like this: ds[{1 -> Query[{"a" -> (999 &)}]}] Multiple fields can be updated simultaneously: ds[{1 -> (<| #, "...


17

Currently in Mathematica v10 you need to be careful joining datasets as the Key[] function doesn't work with a string key for a dataset (it does for an association) . See Taliesin's comment below. Personally I think the help is a bit misleading referring to SQL joins with associations when they are more like "joins" in a NOSQL database. Examples of ...


17

As @Mr.Wizard notes in a comment, some discussion about the overheads associated with querying can be found in another question (56609). This response will use Mathematica version 10.1.0 to examine the specific behaviour described in the present question. The general principles under discussion are the same for the various 10.0.x versions, but some details ...


17

Preface: For everyone: be aware that this behaviour is very likely to change soon (the name of the hook variable flag, etc). In fact, I did change this for 10.2, where Export would fall back to the standard Export in case when the specialised hook fails. Leonid Shifrin I get the same behaviour using Mathematica version 10.1 on Windows 7/64-bit. Dataset has ...


16

This is interesting: If you replace the symbol y with the string "y", Append works fine. Append[ds, <|"a" -> 5, "b" -> "y"|>] Also, if you start out with Symbols, then it works fine: ds = Dataset[{<|"a" -> 1, "b" -> x|>, <|"a" -> 2, "b" -> y|>, <| "a" -> 3, "b" -> z|>, <|"a" -> 4, "b" -> x|>...


16

Let's use the Titanic dataset tit = ExampleData[{"Dataset", "Titanic"}]; Let's see it's columns tit[Union, Keys] Let's choose an objective obj = "survived"; Let's add an id to each row tit = tit[AssociationThread[Range@Length@#, #] &]; Let's create a database that splits the features and objective, to make this general. titSplit = tit[All, &...


16

This is a bug in the 10.0.2 version of the type inferencer, which now goes inside pure functions†. It's 'harmless' in that the type inference will just give up and fall back on deduction (which is what it was going to do anyway). I've fixed this for version 10.0.3, but in the meantime, here's a patch that will prevent the message: Begin["TypeSystem`...


16

The problem with the query proposed in the question is that it is attempting to apply Select to each row. As @Kuba points out in a comment, the use of Select is unnecessary. The query can be expressed like this: ds[All, If[#"VAL1" == #"VAL2" == #"VAL3", <|#, "Type" -> "None"|>, #] &] The same result can be obtained by an alternate ...


16

The problem appears to be due to a v10.2 bug in the Dataset visualization code that generates the box form of a dataset. It is not correctly distinguishing between Dataset being used as a constructor function and Dataset being used as the head of a constructed dataset. It assumes the latter case unconditionally, giving the exhibited messages for the former ...


16

There are two issues under discussion: 1) the distinct dataset visualizations for the same data and 2) ways to update dataset subelements in place. We will discuss these separately. Distinct Dataset Visualizations The way a dataset is displayed is sensitive to the data type of the dataset. That type, in turn, is sensitive to the history of the dataset. ...


15

I have implemented the underlying kernel functionality that is needed to make this possible. However it is not yet implemented on the Dataset side. I don't think this will happen in the immediate future owing to other priorities. Here is a stop-gap that implements a simple version of mutable updating, this of course is not production-grade. I'm happy for ...


15

I think what you want is the following: dataSet[ListContourPlot, {#a, #e, #c} &] dataSet[ListDensityPlot, {#a, #e, #c} &] If you still want to use the Keys directly then you'll have to force ListContourPlot and ListDensityPlot to extract only the Values as follows: dataSet[ListContourPlot[Values@#] &, {"a", "e", "c"}]


15

UPDATE The behaviour no longer occurs in version 10.3. Here are minimal reproduction steps, from a fresh 10.2.0 kernel: Module[{data = {{1}}}, data // Dataset // Transpose] (* data::shdw: Symbol data appears in multiple contexts {Streaming`LazyList`Testing`,Global`}; definitions in context Streaming`LazyList`Testing` may shadow or ...


Only top voted, non community-wiki answers of a minimum length are eligible