# Error changing Dataset using Part

Update

It was a bug in the documentation of V10.0, this functionality was not implemented yet, and V10.1 changed the documentation, removing it. It's a pity, because it's a very useful operations, common in other languages like R. I miss data.frame like notation in Mathematica.

In the new guide Computation With Structured Datasets we can find this part, on how to change a Dataset

But if we create a Dataset like:

ds=Dataset[{<|"a"->1,"b"->"x"|>,<|"a"->2,"b"->"y"|>,<|"a"->6,"b"->"z"|>}];


And then make:

ds[[1, 1]] = 2


Or, closer to my real case test:

ds[[All, "a"]] = Accumulate@Normal@ds[[All, "a"]]


We get an error:

"Part specification ds[[1,1]] is longer than depth of object"

"Part specification ds[[All,1]] is longer than depth of object. "

Is this a Bug?

Setting is not working on Dataset as stated by documentation.

This post on Wolfram Community

• Unfortunately not in V10.0.1 yet... – Murta Sep 17 '14 at 1:42
• StringReplace[%,"V10.0.1"-> "V10.0.2"] – Murta Dec 11 '14 at 1:35
• StringReplace[%%,"V10.0.1"-> "V10.1.0"] – Murta Mar 30 '15 at 21:01
• This is no longer documented to work as of 10.1.0. As Tali mentions in his answer below, the inclusion of this comment in the original documentation was erroneous. – Stefan R Jun 8 '15 at 15:07
• @StefanR I know about that. But this would be a nice way to handler data, and should be considered in the future. In R, it's a very natural way to do Data Frame manipulations. – Murta Jun 8 '15 at 18:41

I'm the developer of Dataset.

Yes, this is a gross documentation oversight. We planned this functionality but had to push it back to a point release. Somehow no-one caught this piece of legacy documentation.

I've filed a bug on the documentation problem right now, it's easy to fix.

As for when L-value assignment will be available, I'm hoping 10.0.1 or 10.0.2, which are in the next month or two. It gets complicated, because you might well want to write things like:

dataset[ Select[#age > 30&] , "salary"] *= 2


That's certainly a powerful kind of operation, but also hard to implement. Even part-like assignments can get complicated when you are assigning multidimensional datasets to each other.

Thanks for trying the functionality, though!

• Tks for your clarification. I'll wait for it, it's a very useful operation and I'm happy that I won't need to wait for V11. Using this opportunity, have you saw this post in Wolfram Community about Dataset memory consumption? There are plans to efficient Tabular Data in V10? – Murta Jul 10 '14 at 19:31
• @Murta Yes, moving to column-oriented will make things much better. But before I could do that I had to lay the groundwork in the form of a type system that could represent the "logical shape", even if the "physical layout" is different. And of course Leonid is working on making this whole process scale to out-of-core computation against data that lives on disk. – Taliesin Beynon Jul 10 '14 at 20:26
• @TaliesinBeynon your example is quite funny ! – faysou Nov 17 '14 at 8:46
• I'm wondering what's the status of the problem. Looks like it's not in V11.1. – xslittlegrass Apr 1 '17 at 16:47
• @xslittlegrass Mr. Wizard's answer is correct. With a lot of optimization work we could make Dataset opportunistically store tables in column-oriented form. Indeed that's always been the plan. But that will take months to implement properly and currently my priorities lie with neural networks. – Taliesin Beynon Apr 5 '17 at 16:59

In lieu of Set, the Query syntax offers various ways to update selective elements of a dataset. For example, we can change the value of the field a in the first row like this:

ds[{1 -> (<| #, "a" -> 999|> &)}]


or like this:

ds[{1 -> Query[{"a" -> (999 &)}]}]


Multiple fields can be updated simultaneously:

ds[{1 -> (<| #, "a" -> 999, "b" -> "ZZZ" |> &)}]


We can update selective rows, in this case field "b" in rows with even a:

ds[All, If[EvenQ[#a], <| #, "b" -> "!!!!"|>, #] &]


The accumulation use case can be accomplished like this:

With[{a = ds[Accumulate, "a"]}
, ds @ MapIndexed[<| #, "a" -> a[[First@#2]] |> &]
]


or like this:

Module[{acc = 0}, ds[All, {"a" -> (acc += # &)}]]


Note that none of these operations involve destructively altering the dataset, so they should all read ds = ds[...] if desired. Presumably Set will eventually perform destructive updates in those restricted circumstances that Mathematica tolerates mutation.

• Examples such as these are sure to increase the fun factor for the WRI employees working to compile the Query language into SQL ;) – WReach Jul 27 '14 at 0:11
• Nice examples. +1. – Murta Jul 27 '14 at 0:15
• @WReach, are WRI employees working to compile the Query language into SQL? Will this be brought into DatabaseLink? – ArgentoSapiens Nov 6 '14 at 16:24
• @ArgentoSapiens I have no current information about this. My glib comment was based upon the fact that pre-release versions of the V10 documentation contained extensive references to such capability. Those references were withdrawn very late, just before the official V10 release. I speculate that the functionality under discussion in the question would (or did) prove to be challenging to support across multiple back-end technologies. – WReach Nov 6 '14 at 16:55

Though I don't know what is the efficiency impact of it, a workaround could be converting the Dataset to Association by Normal, making the update on the Association, then converting it back to Dataset.

ds = Dataset[{<|"a" -> 1, "b" -> "x"|>, <|"a" -> 2, "b" -> "y"|>, <|"a" -> 6, "b" -> "z"|>}]

ds = Module[{temp = Normal[ds]},
temp[[All, "a"]] = Accumulate[temp[[All, "a"]]];
temp // Dataset]


I have implemented the underlying kernel functionality that is needed to make this possible. However it is not yet implemented on the Dataset side. I don't think this will happen in the immediate future owing to other priorities.

Here is a stop-gap that implements a simple version of mutable updating, this of course is not production-grade. I'm happy for anyone who wants to modify this answer to extend its functionality, add error handling, etc.

Unprotect[Dataset];
LanguageSetMutationHandler[Dataset, DatasetMutationHandler];

SetAttributes[DatasetMutationHandler, HoldAllComplete];
DatasetMutationHandler[Set[sym_Symbol[[args___]], newvalue_]] := Block[{tmp},
tmp = Normal[sym];
tmp[[args]] = If[DatasetValidDatasetQ[newvalue], Normal[newvalue], newvalue];
sym = Dataset[tmp];
];


You can use it as follows:

In[51]:= d = Dataset[{1, 2, 3}];
d[[2 ;; 3]] = 99;
Normal[d]

Out[53]= {1, 99, 99}

• I see that this function was added in 10.4. Is it already reliable and usable there? (I do not have a use for it at this moment, just asking for the future.) – Szabolcs Apr 5 '17 at 18:35
• @Szabolcs yes, it is. it's used in production to implement CloudExpression. you may notice however that Language`HasMutationHandlerQ returns the opposite of the correct answer, but it's not a very important function. – Taliesin Beynon Apr 6 '17 at 0:25