165

I. General I will first try to briefly answer the questions, and then illustrate this with a small but practical application. 1.Speed of insertion / deletion Associations are based on so called Hash Array Mapped Trie persistent data structure. One can think of this as a nested hash table, but it is more than that, because it has the following properties: ...


78

Here are a few ways, each of which operates upon the individual component associations. In the following discussion, recall that when a key name is not a valid symbol we can write, for example, #["col_name"] instead of #col. We can explicitly construct a new association that includes all of the old columns and adds a new one: ds[All, <| "col1"->"...


49

A Dataset represents an abstraction over a structured collection of data. Notionally, it is restricted to "well-behaved" data -- data that comes in simple forms that can be readily interchanged with external systems such as relational databases, XML documents, JSON documents, etc. These are commonplace forms such as vectors, records ("structs"), tuples, ...


48

I helped design Association, and I designed and implemented Dataset, so I wanted to comment on question 3: Dataset is designed explicitly for hierarchical data. It supports any 'shape' of data, inferring the shape when the Dataset is first created. It also tracks the shape of the data as transformations are applied to the dataset, using a type-inference ...


34

I'm the developer of Dataset. Yes, this is a gross documentation oversight. We planned this functionality but had to push it back to a point release. Somehow no-one caught this piece of legacy documentation. I've filed a bug on the documentation problem right now, it's easy to fix. As for when L-value assignment will be available, I'm hoping 10.0.1 or 10....


31

Actually, Mathematica has this stuff built in. I couldn't find this information anywhere, so posting it here for general reference. You can use it like this: Needs["Parallel`Queue`Priority`"] Unprotect@Priority; Priority[i_Integer] := Abs[i] q = priorityQueue[]; EnQueue[q, 10]; EnQueue[q, 7]; EnQueue[q, -20]; Size[q] == 3; Top[q] == -20; Normal[q] == {-20, ...


31

Such functions set up a one-time iterator, i.e. a GeneralUtilities`Iterator. Its functionality may be inspected with ? GeneralUtilities`Iterator and so forth. Except for a ConstantIterator which always returns the same value (forever), they go sequentially through values until they return GeneralUtilities`IteratorExhausted. The main (if internal) utility ...


29

Another way that works (for one or more columns) is: ds[All, <|#, "col3" -> #col1 + #col2, "col4" -> #col1 - #col2|> &] This gives: Also, sometimes the values for the new column might not be straightforwardly computed row by row. For example, you might have calculations like this: newcol = RotateLeft @ Normal[ds[All, (#col1 + #col2 &)...


28

There is a ToAssociations function in the GeneralUtilities` package that is perfect for this and for converting nested JSON rules to associations: Needs["GeneralUtilities`"] ToAssociations@rdata (* <|"a" -> <|"b" -> 1, "c" -> {7, 8, 9}|>, "d" -> 3|> *) This preserves the inner most list that is not a list of rules. As for your ...


28

Yes. Use System`Private`SetNoEntry on any expression which you want to protect in this manner. This works on per-expression basis, so you have to apply this function to any instance which you want to protect. The result is a reference to the same expression. The changes are performed in-place (no copy is created): expr = h[1, 2, 3] (* h[1, 2, 3] *) ...


25

I would like to point out that Listable in a pure Function effectively unpacks the array, and makes it much slower than Map for pure Functions. Downvalues always unpack so SetAttributes[f, Listable] doesn't affect performance there. The bottom line is that if one wants to use user defined listability it must be inside a compiled function, otherwise use Map ...


24

First of all, it's not a new data structure, it's a new only in a Wolfram Mathematica. About complexity of a data structure. It's a Wolfram implementation of a hash-map. With a complexity of an operations (worst case in parenthesis): Space O(n) Search O(1) (O(n)) Insert O(1) (O(n)) Delete O(1) (O(n)) This can be easily checked: create custom ...


22

You can implement an imperative-style circular buffer. big = Range@1*^7; size = Length@big; pointer = size; updateElement[new_Integer] := (pointer = 1 + Mod[pointer, size]; big[[pointer]] = new) Do[updateElement[RandomInteger@99], {100}] // AbsoluteTiming {0.000374, Null} To bring the buffer back to the normal form use big = RotateLeft[big, Mod[...


22

Update: Mathematica version 10 introduced the builtin Association data structure (see tutorial and answer below), which has most important features of dictionaries. Mathematica has no obvious hash-table structure but what most people forget is, that the DownValues of symbols, which means the simple, always-used function definitions, are implemented using ...


21

This is going to be transcript of Roman E. Maeder's priority queue code with any updates I can find to make to take advantage of functions added since he wrote it. I believe I am within right to copy it here for noncommercial purposes. Refactor v0.2 -- any bugs are almost certainly my own. BeginPackage["PriorityQueue`"] MakeQueue::usage = "MakeQueue[...


19

I arrived very late to this party and I'm very much afraid that nobody comes here anymore. Still I'm posting this in hope that an occasional visitor may find it a practical approach to implementing data structures with named fields within Mathematica. The concept The idea is to use protected symbols to name a structure and its fields. The symbol that names ...


18

Preamble I will discuss here two methods for doing computations on very large data sets which don't fit into memory. The first method is based on sequential reading of chunks of data from a file. The second method is based on converting a data set to a file-backed list representation. The unifying idea for both methods is the use of iterators as a useful ...


18

As @Mr.Wizard notes in a comment, some discussion about the overheads associated with querying can be found in another question (56609). This response will use Mathematica version 10.1.0 to examine the specific behaviour described in the present question. The general principles under discussion are the same for the various 10.0.x versions, but some details ...


18

A level specification can help (we have to specify the head to thread over, too): Thread[{{a, b}, {{x, y}, {z, w}, {u, v}}}, List, {2}] {{{a, b}, {x, y}}, {{a, b}, {z, w}}, {{a, b}, {u, v}}} Note that this unpacks arrays, so I would not suggest that for big datasets. I that case, I would propose a ConstantArray/Transpose combo: n = 1000000 ; a = ...


17

In lieu of Set, the Query syntax offers various ways to update selective elements of a dataset. For example, we can change the value of the field a in the first row like this: ds[{1 -> (<| #, "a" -> 999|> &)}] or like this: ds[{1 -> Query[{"a" -> (999 &)}]}] Multiple fields can be updated simultaneously: ds[{1 -> (<| #, "...


16

Making discrete Data an abstract data type I would try to do something like the following: (* write a constructor function for the data type 'discrete data' *) discreteData[ scale_?NumericQ , bias_?NumericQ , tally : { { _?NumericQ , _?NumericQ } .. } ] := discreteData @ Association[ "scale" -> scale, "bias" -> bias, "...


16

This way of comparison is not directly exposed in top-level at the moment. However, it could be done through the compiler, for example sameInstanceQ = FunctionCompile[Function[{Typed[e1, "Expression"], Typed[e2, "Expression"]}, Native`SameInstanceQ[e1, e2]]]; a = CreateDataStructure["Value", 1]; b = a["Copy"]; c = b; SameQ[a, b, c] (* True *) ...


15

This is my favorite way to draw Pascal's triangle. pt = NestList[{0, ##} + {##, 0} & @@ # &, {1}, #] &; ptform[pt : {_List ..}] := With[{n = Length@Last@pt, long = Max@Map[StringLength@ToString@# &, pt, {2}]}, Graphics[MapIndexed[Text[#, {#2 - #/2, -#} & @@ #2] &, pt, {2}], PlotRange -> All, AspectRatio -> 0.7, ...


15

I have implemented the underlying kernel functionality that is needed to make this possible. However it is not yet implemented on the Dataset side. I don't think this will happen in the immediate future owing to other priorities. Here is a stop-gap that implements a simple version of mutable updating, this of course is not production-grade. I'm happy for ...


14

Mathematica 10 has introduced Association, which has most of the important properties of a dictionary data structure. someData = <| "name" -> "Bob", "age" -> 23 |> In[1]:= someData["name"] Out[1]= Bob In[2]:= someData["age"] Out[2]= 23 In[3]:= someData[[2]] Out[3]= 23 For more info, see Association, DataSet (which builds on Association) ...


14

Though I don't know what is the efficiency impact of it, a workaround could be converting the Dataset to Association by Normal, making the update on the Association, then converting it back to Dataset. ds = Dataset[{<|"a" -> 1, "b" -> "x"|>, <|"a" -> 2, "b" -> "y"|>, <|"a" -> 6, "b" -> "z"|>}] ds = Module[{temp = Normal[...


14

The basic reason is that once you convert a tensor expression into a SparseArray, you've "given control" of all levels of that expression to SparseArray to manage on your behalf in an efficient way (the number of levels is the rank of the tensor, to mix jargon). SparseArray will then try to maintain the illusion that those levels are still really there. ...


14

Update 2 Per request, I extended this to handle arbitrary sizes and rotations. It was a huge hassle to figure out how to get the appropriate permutations for the individual rotations for arbitrary sized cubes, but it worked out. Here's what it looks like: r1 = RubiksCube["Size" -> 4]; r1@"Colors" = ColorData["Atoms"] /@ {6, ...


14

It never updates for me properly. This is what I do: ro = ResourceObject["Epidemic Data for Novel Coronavirus COVID-19"]; DeleteObject[ro]; ro = ResourceObject["Epidemic Data for Novel Coronavirus COVID-19"]; Just delete and re-download. Wolfram basically just provides some minor convenience by re-formatting the data to a more Mathematica-compatible ...


13

A way of getting around the a_/;test[a] syntax is to write out the tests in string form, and use replace to insert the values. For this to work you need to build rules from your table. Here is a simple implementation: SetAttributes[queryCriteria, HoldAll] queryCriteria[theTable_, query_] := Function[{entry}, Unevaluated[query] /. (Rule @@@ Transpose[{...


Only top voted, non community-wiki answers of a minimum length are eligible