# Most efficient way to replace/ update the values multiple rows and columns of a Dataset

Suppose we wish to replace the cols "b" and "c" of rows 3 to 5 in the following dataset to b and c respectively:

dataset = Dataset[{
<|"a" -> 1, "b" -> "x", "c" -> {1}|>,
<|"a" -> 2, "b" -> "y", "c" -> {2, 3}|>,
<|"a" -> 3, "b" -> "z", "c" -> {3}|>,
<|"a" -> 4, "b" -> "x", "c" -> {4, 5}|>,
<|"a" -> 5, "b" -> "y", "c" -> {5, 6, 7}|>,
<|"a" -> 6, "b" -> "z", "c" -> {}|>}]

1. What is the most efficient way to do it at scale, i.e. assuming the replacements are going to be across multiple/ all rows in large datasets.
2. Is is possible to do it inline, i.e., in the current dataset, without having to recreate a new dataset?
3. It would be preferrable if the updates are computed (i.e., not constants as shown in this toy example)
• It's not really an answer but two things I would mention: 1) I've found Dataset to not be very handy for working with data. I tend to process data as associations and only create a Dataset at the end when I'm going to do queries / analysis. 2) Associations support structural sharing so #2 may not actually be a concern (see youtu.be/0ZnZlAKMZdg?t=843 ) -- obviously I don't know everything about what you're doing but if it's just about memory usage then it should be ok.
– Dan
Dec 10, 2018 at 2:51

You can use ReplacePart directly on the dataset but it seems a tedious/inefficient way of doing this. It is scalable as long as you can use patterns or Alternative to select the positions you want to replace. I expect someone else will be able to provide a more efficient method that works directly on the dataset without first converting to a list of associations.

dataset=ReplacePart[dataset, {{3 | 4 | 5, "b"} -> b, {3 | 4 | 5, "c"} -> c}]


• This does not perform an in-place modification. If you evaluate dataset after the ReplacePart, it is unchanged. Dec 10, 2018 at 1:01
• @RohitNamjoshi good point. I'll leave the answer up even though it does not provide in-place modification Dec 10, 2018 at 5:21

I mentioned it in the comments but basically I like to keep it pretty simple and while I'm not sure if it is the fastest way to do it it is nice and clean and the memory usage is not as bad as it would seem due to Mathematica's good choice of persistent data structures to build Associations on.

In[1]:= MemoryInUse[] / 1000000 // N
Out[1]= 96.5472
In[2]:= data = ResourceData["US State Income"]
In[3]:= MemoryInUse[] / 1000000 // N
Out[3]= 117.604
(* Updating a field *)
In[4]:= data2 = Dataset[<|"State"->#State, "Income"->#Income/2.0|>&/@Normal@data]
In[5]:= MemoryInUse[] / 1000000 // N
Out[5]= 117.885
(* Appending a field *)
In[6]:= data3 = Dataset[<|#, "IncomeSquared"->#Income^2|>&/@Normal@data]
In[7]:= MemoryInUse[] / 1000000 // N
Out[7]= 118.574


Actually, I don't now how to do that with a Dataset but with nested Lists and Association, in-place modification is possible:

assoc = Normal[dataset];
assoc[[3 ;; 5, {"b", "c"}]] = ConstantArray[{b, c}, 3];
assoc // Dataset


So, if you don't need any specific properties of Dataset, I'd suggest to stick with nested Associations.

• Im trying to manipualte a real-world dataset and am exploring using Dataset in MMA for its GroupBy and other convenience features like plotting by columns etc. Dec 9, 2018 at 18:10