Hot answers tagged

80

Here are a few ways, each of which operates upon the individual component associations. In the following discussion, recall that when a key name is not a valid symbol we can write, for example, #["col_name"] instead of #col. We can explicitly construct a new association that includes all of the old columns and adds a new one: ds[All, <| "col1"->"...


49

A Dataset represents an abstraction over a structured collection of data. Notionally, it is restricted to "well-behaved" data -- data that comes in simple forms that can be readily interchanged with external systems such as relational databases, XML documents, JSON documents, etc. These are commonplace forms such as vectors, records ("structs"), tuples, ...


39

Official Statement In case some of you missed it: There was an official reply from one of the WRI devs recently in our chat Hi. This is Jose, from Wolfram. We are aware of an unacceptable slowdown in some Dataset expressions, due to a bad dynamic interaction with the summary boxes of some objects. Both the TimeSeries objects of the coronavirus datasets ...


38

Theoretically, Dataset supports any number of columns. The behavior you are seeing is actually because the type deduction that Dataset is doing behind the scenes isn't perfect (and indeed in some sense cannot be perfect). Your synthetic example is such that your second list of associations is "most consistent" with a particular type that doesn't typeset as ...


38

This response defines a function called traceTypes which provides a quick-and-dirty visualization of type system operation. The function is somewhat fragile as it depends upon undocumented implementation details in version 10.2. Despite this fragility, it might be useful for study purposes as it handles many common type system use cases. The code for the ...


34

I'm the developer of Dataset. Yes, this is a gross documentation oversight. We planned this functionality but had to push it back to a point release. Somehow no-one caught this piece of legacy documentation. I've filed a bug on the documentation problem right now, it's easy to fix. As for when L-value assignment will be available, I'm hoping 10.0.1 or 10....


30

If you are willing to use an undocumented and future-incompatible hack you can set the global variable Dataset`FormattingCompound`PackagePrivate`$Limit to larger than its default value of 64 to see more of the dataset. For example, Dataset[RandomInteger[10, {1000, 5}]] shows 16 rows by default, but by doubling $Limit to 128 you will see 32 rows, etc. I will ...


29

Another way that works (for one or more columns) is: ds[All, <|#, "col3" -> #col1 + #col2, "col4" -> #col1 - #col2|> &] This gives: Also, sometimes the values for the new column might not be straightforwardly computed row by row. For example, you might have calculations like this: newcol = RotateLeft @ Normal[ds[All, (#col1 + #col2 &)...


28

Until scrollbars are implemented... ... I have found this little function for simple tabular data to work out fine with everyday work. One can size the viewer window (and probably use the tricks posted so far to extend the range of formatted output?). Also you can set a specific column (numerically) to be a repeated column, eg. to remain fixed while the ...


26

data = N @ Normalize[#, Total] & @ Counts @ Characters @ ExampleData[ {"Text", "DeclarationOfIndependence"} ]; Dataset @ data Dataset[KeyValueMap[<|"char" -> #, "freq" -> #2|> &, data]]


26

Dataset was restructured in the 12.1 release in order to support expanded formatting options and interactivity such as hiding and sorting. As a result, some Dataset outputs showed a slowdown due to inefficiencies in the dynamic output structures they produce. Because the code of Dataset is automatically field upgradable, we have released an update to its ...


24

Inspired by WReach's answer, I started playing with Query based approach and here's what I came up with: data = {"days" -> {1, 2, 6, 8}, "area" -> {3, 6, 8, 2}, "frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}} With the data in the above form, we just create a Dataset simply as follows: dataset = Dataset[data]; Don't worry ...


23

This is a corner case that occurs when ascending operators are interleaved between descending operators. This case falls into an undocumented grey area. In the Details and Options section of the Query documentation, we read the following special rule: When one or more descending operators are composed with one or more ascending operators (e.g. desc /* ...


23

One simple way of "filtering" your data is to treat the points as a graph, and search for the shortest path from left to right: xScale = 10.; xy = Transpose[{N[Range[Length[data]]]*(xScale/Length[data]), data}]; start = {-xScale, Mean[data]}; finish = {2*xScale, Mean[data]}; graph = NearestNeighborGraph[Join[{start}, xy, {finish}], 25]; graph = ...


22

We can explicitly construct a new association with key names of our choosing: dataset[All, <| "a" -> "a", "h" -> "b", "c" -> "c" |>] Alternatively, a function could be applied to the keys: dataset[All, KeyMap[# /. "b" -> "h" &, #] &] Note that a bug in ...


19

I was playing around, trying to come up with a purely query-based solution. The result is by no means natural, but perhaps it holds some academic interest or will spark some insight into a better answer: Dataset[ { "days" -> {1, 2, 6, 8} , "area" -> {3, 6, 8, 2} , "frequency" -> {1, 4, 4, 2} , "height" -> {2, 3, 11, 6} } ][ Transpose, ...


19

We can start by importing the file as an XMLObject: $url = "https://dl.dropboxusercontent.com/u/1012958/iTunes%20Library.xml"; $xml = Import[$url, {"XML", "XMLObject"}]; Short[$xml, 4] (* XMLObject[Document][ { XMLObject[Declaration][Version->1.0,Encoding->UTF-8] , XMLObject[Doctype][plist,Public->-//Apple Computer//DTD PLIST 1.0//EN,&...


18

As @Mr.Wizard notes in a comment, some discussion about the overheads associated with querying can be found in another question (56609). This response will use Mathematica version 10.1.0 to examine the specific behaviour described in the present question. The general principles under discussion are the same for the various 10.0.x versions, but some details ...


17

In lieu of Set, the Query syntax offers various ways to update selective elements of a dataset. For example, we can change the value of the field a in the first row like this: ds[{1 -> (<| #, "a" -> 999|> &)}] or like this: ds[{1 -> Query[{"a" -> (999 &)}]}] Multiple fields can be updated simultaneously: ds[{1 -> (<| #, "...


17

Currently in Mathematica v10 you need to be careful joining datasets as the Key[] function doesn't work with a string key for a dataset (it does for an association) . See Taliesin's comment below. Personally I think the help is a bit misleading referring to SQL joins with associations when they are more like "joins" in a NOSQL database. Examples of ...


17

Preface: For everyone: be aware that this behaviour is very likely to change soon (the name of the hook variable flag, etc). In fact, I did change this for 10.2, where Export would fall back to the standard Export in case when the specialised hook fails. Leonid Shifrin I get the same behaviour using Mathematica version 10.1 on Windows 7/64-bit. Dataset has ...


17

There are two issues under discussion: 1) the distinct dataset visualizations for the same data and 2) ways to update dataset subelements in place. We will discuss these separately. Distinct Dataset Visualizations The way a dataset is displayed is sensitive to the data type of the dataset. That type, in turn, is sensitive to the history of the dataset. ...


16

This is interesting: If you replace the symbol y with the string "y", Append works fine. Append[ds, <|"a" -> 5, "b" -> "y"|>] Also, if you start out with Symbols, then it works fine: ds = Dataset[{<|"a" -> 1, "b" -> x|>, <|"a" -> 2, "b" -> y|>, <| "a" -> 3, "b" -> z|>, <|"a" -> 4, "b" -> x|>...


16

Let's use the Titanic dataset tit = ExampleData[{"Dataset", "Titanic"}]; Let's see it's columns tit[Union, Keys] Let's choose an objective obj = "survived"; Let's add an id to each row tit = tit[AssociationThread[Range@Length@#, #] &]; Let's create a database that splits the features and objective, to make this general. titSplit = tit[All, &...


16

This is a bug in the 10.0.2 version of the type inferencer, which now goes inside pure functions†. It's 'harmless' in that the type inference will just give up and fall back on deduction (which is what it was going to do anyway). I've fixed this for version 10.0.3, but in the meantime, here's a patch that will prevent the message: Begin["TypeSystem`...


16

The problem with the query proposed in the question is that it is attempting to apply Select to each row. As @Kuba points out in a comment, the use of Select is unnecessary. The query can be expressed like this: ds[All, If[#"VAL1" == #"VAL2" == #"VAL3", <|#, "Type" -> "None"|>, #] &] The same result can be obtained by an alternate ...


16

The problem appears to be due to a v10.2 bug in the Dataset visualization code that generates the box form of a dataset. It is not correctly distinguishing between Dataset being used as a constructor function and Dataset being used as the head of a constructed dataset. It assumes the latter case unconditionally, giving the exhibited messages for the former ...


16

Preamble I will try to illustrate several possibilities. Some of the following will be purely "manual" manipulations based on some top-level code of the type one would write. Test data for SQL and HDF methods, and a tabular format For SQL and HDF methods, I will use a simple tabular format for the data: {columnNames:{__String}, data:{__List}} which is, ...


16

Let's take some example lists: lis1 = {1, 2, 3, 4, 5}; lis2 = {6, 7, 8, 9, 0}; lis3 = {2, 4, 6, 8, 9}; Now you can combine your lists into one list and your headers into a different list: data = {lis1, lis2, lis3}; header = {"a", "b", "c"}; To create a dataset just do: Dataset@Map[AssociationThread[header, #] &]@Transpose[data]


16

Actully we have a easiest way,suppose you have dataset like dataset=Dataset[{{"a", 10}, {"b", 11}, {"c", 12}, {"d", 5}, {"e", 99}}] You can add a column name dataset[All, <|"char" -> 1, "freq" -> 2|>] Performance But if you have a large data set,I have compared FIVE solution ...


Only top voted, non community-wiki answers of a minimum length are eligible