# Tag Info

165

I. General I will first try to briefly answer the questions, and then illustrate this with a small but practical application. 1.Speed of insertion / deletion Associations are based on so called Hash Array Mapped Trie persistent data structure. One can think of this as a nested hash table, but it is more than that, because it has the following properties: ...

80

Here are a few ways, each of which operates upon the individual component associations. In the following discussion, recall that when a key name is not a valid symbol we can write, for example, #["col_name"] instead of #col. We can explicitly construct a new association that includes all of the old columns and adds a new one: ds[All, <| "col1"->"...

58

Preamble This is a very good question, because answering it will make it very clear what immutability means, both in general and in the context of Associations. General A few general words on immutability Associations are immutable data structures. This means that they carry no state, and a copy of an Association is another completely independent Association....

49

I helped design Association, and I designed and implemented Dataset, so I wanted to comment on question 3: Dataset is designed explicitly for hierarchical data. It supports any 'shape' of data, inferring the shape when the Dataset is first created. It also tracks the shape of the data as transformations are applied to the dataset, using a type-inference ...

49

A Dataset represents an abstraction over a structured collection of data. Notionally, it is restricted to "well-behaved" data -- data that comes in simple forms that can be readily interchanged with external systems such as relational databases, XML documents, JSON documents, etc. These are commonplace forms such as vectors, records ("structs"), tuples, ...

30

I hesitate to add anything after @Leonid's comprehensive answer, but I'd like to point out that an easy way to achieve the stated goal is to define f like this: f[x_] := <| x, "isFirstValueTrue" -> x@"firstValue" |> ... which yields the desired result when mapped across the associations in x: f /@ x (* { <|"firstValue" -> True, "...

29

Another way that works (for one or more columns) is: ds[All, <|#, "col3" -> #col1 + #col2, "col4" -> #col1 - #col2|> &] This gives: Also, sometimes the values for the new column might not be straightforwardly computed row by row. For example, you might have calculations like this: newcol = RotateLeft @ Normal[ds[All, (#col1 + #col2 &)...

29

Initial data: peopleFacts = <| alice -> <|age -> 29, shoeSize -> 7|>, bob -> <|age -> 27, sex -> male, hair -> <|Color -> RGBColor[1, 0, 0]|> |> |>; Here is a version of RecurAssocMerge reduced to a single definition. MergeNested = If[MatchQ[#, {__Association}], Merge[#, #0], Last[#]] & ...

28

There is a ToAssociations function in the GeneralUtilities package that is perfect for this and for converting nested JSON rules to associations: Needs["GeneralUtilities"] ToAssociations@rdata (* <|"a" -> <|"b" -> 1, "c" -> {7, 8, 9}|>, "d" -> 3|> *) This preserves the inner most list that is not a list of rules. As for your ...

28

General If you are not going to change your list of rules after you construct them, Dispatch is pretty good. From the user's viewpoint, the main difference is that it is cheap to add new key-value pairs to associations, or remove existing ones, constructing new associations. Not so with Dispatch - once you obtained the Dispatch-ed set of rules, you can't ...

26

data = N @ Normalize[#, Total] & @ Counts @ Characters @ ExampleData[ {"Text", "DeclarationOfIndependence"} ]; Dataset @ data Dataset[KeyValueMap[<|"char" -> #, "freq" -> #2|> &, data]]

24

First of all, it's not a new data structure, it's a new only in a Wolfram Mathematica. About complexity of a data structure. It's a Wolfram implementation of a hash-map. With a complexity of an operations (worst case in parenthesis): Space O(n) Search O(1) (O(n)) Insert O(1) (O(n)) Delete O(1) (O(n)) This can be easily checked: create custom ...

22

As Kirill Belov notes in a comment, the issue is related to the fact that the list a1 is a packed array (generated by Range) whereas the list a2 is not packed. Count, Position and Depth unexpectedly act as if the packed array is atomic. This is very likely a bug since 1) the expected behaviour occurs if the top-level expression is a list instead of an ...

21

One can use TracePrint to see how Flatten is being used by Merge: TracePrint[ Merge[{assoc,assoc}, Flatten], _Flatten, TraceAction->Print@*DeveloperPackedArrayForm ]; Flatten[{PackedArray[Real,<10>],PackedArray[Real,<10>]}] Flatten[{PackedArray[Real,<10>],PackedArray[Real,<10>]}] Flatten[{PackedArray[Real,<10>],...

20

Adding the same key to an Association will replace the previous value, which leads to this solution: assoc = <| "id" -> 3, "freq" -> 4 |>; <|#, "freq" -> #freq + 1|> &@assoc (* <| "id" -> 3, "freq" -> 5 |> *) The following alternatives also work: assoc["freq"] = 5 assoc["freq"] = assoc["freq"] + 1 assoc["freq"] += 1

19

Somewhat similar to the answer of evanb, but without explicit mutations: keyRename[a_, old_ -> new_] /; KeyExistsQ[a, old] := KeyDrop[old]@Append[a, new -> a[old]] So that keyRename[assoc, "this_key_is_too_long_to_type" -> "c"] (* <|"a" -> 1, "b" -> "x", "c" -> {1}|> *) It should be noted that this solution doesn't preserve the ...

19

PositionIndex[assoc][<value>] For example: assoc = <|a -> 1, b -> 2, c -> 3, d -> 1, e -> 4, f -> 1|>; PositionIndex[assoc][1] {a, d, f} And PositionIndex[assoc] <|1 -> {a, d, f}, 2 -> {b}, 3 -> {c}, 4 -> {e}|> Alternatively: PositionIndex[assoc] // Lookup[4] {e}

19

Looks like a bug to me as well. Just as Michael I would suspect that the problem is more with Lookup than Block or Associations in general. Here is a code example which seems to confirm that it is Lookup which does some caching and obviously doesn't take into account that the Blocked variables value has changed: ClearAll@test test[var_String] := Block[{...

19

Preamble The real problem here seems somewhat deeper than what the (mostly correct) observations in comments indicate. In Mathematica, a number of objects, which are so-called raw objects (including in particular Association, SparseArray, and Rational) require a non-trivial construction stage, which usually happens during evaluation. Had Mathematica's ...

18

Association is atomic: <|x -> 1|> // AtomQ True Therefore standard pattern matching inside the structure will not work. You can still match on the implicit head using: MatchQ[<|x -> 1|>, _Association] True There is also AssociationQ: <|x -> 1|> // AssociationQ True MatchQ[<|x -> 1|>, _?AssociationQ] True I ...

18

The undocumented function ToAssociations in the GeneralUtilities package does this In[3]:= Needs["GeneralUtilities"] In[4]:= ToAssociations[lst] (* Out[4]= <|a -> <|1 -> {A1, A2, A3, A4}, b -> 2|>, c -> <|3 -> C3, 4 -> C4|>|> *) As with any undocumented function, use with caution, as contents tend to shift during ...

18

As @Mr.Wizard notes in a comment, some discussion about the overheads associated with querying can be found in another question (56609). This response will use Mathematica version 10.1.0 to examine the specific behaviour described in the present question. The general principles under discussion are the same for the various 10.0.x versions, but some details ...

18

It appears GroupBy does not suffer from this performance issue, so here is an alternative implementation using it, compared to Merge: myMerge[list_, fn_] := GroupBy[Catenate @ Normal[list], Keys -> Values, fn] SeedRandom[1] ascList = Table[<|a -> Random[], b -> Random[]|>, {20000}]; Merge[ascList, Total] // RepeatedTiming myMerge[ascList, ...

16

There is, of course: assoc = <|"a" -> 1, "b" -> "x", "this_key_is_too_long_to_type" -> {1}|>; assoc["c"] = assoc["this_key_is_too_long_to_type"]; assoc["this_key_is_too_long_to_type"] =. assoc (* <|"a" -> 1, "b" -> "x", "c" -> {1}|> *) Not sure if there's an elegant way to do it in one step.

16

This is a bug in the 10.0.2 version of the type inferencer, which now goes inside pure functions†. It's 'harmless' in that the type inference will just give up and fall back on deduction (which is what it was going to do anyway). I've fixed this for version 10.0.3, but in the meantime, here's a patch that will prevent the message: Begin["TypeSystem`...

16

The problem with the query proposed in the question is that it is attempting to apply Select to each row. As @Kuba points out in a comment, the use of Select is unnecessary. The query can be expressed like this: ds[All, If[#"VAL1" == #"VAL2" == #"VAL3", <|#, "Type" -> "None"|>, #] &] The same result can be obtained by an alternate ...

16

Actully we have a easiest way,suppose you have dataset like dataset=Dataset[{{"a", 10}, {"b", 11}, {"c", 12}, {"d", 5}, {"e", 99}}] You can add a column name dataset[All, <|"char" -> 1, "freq" -> 2|>] Performance But if you have a large data set,I have compared FIVE solution ...

15

This operation is performed by Merge: Merge[as, Identity] <|A -> {{1, 11}, {2, 21}, {3, 31}}, B -> {{1, 12}, {2, 22}, {3, 32}}, C -> {{1, 13}, {2, 23}, {3, 33}}|>

15

Mathematica 10.1 almost supports this operation directly: assoc // Query[Transpose] (* <| "a" -> <|"1" -> "x", "2" -> Missing["KeyAbsent", "a"]|>, "b" -> <|"1" -> "y", "2" -> "z"|>, "c" -> <|"1" -> Missing["KeyAbsent", "c"], "2" -> "k"|> |> *) All that remains is to delete the unwanted Missing ...

15

Update Created an upsert function to update/insert new keys and values into a nested association structure. It automatically inserts nested associations where they do not exists and does not need to be assigned back to the original association. It updates existing keys when they are found. ClearAll[upsert] Attributes[upsert] = {HoldFirst}; upsert[dat_?...

Only top voted, non community-wiki answers of a minimum length are eligible