Suppose we wish to replace the cols "b" and "c" of rows 3 to 5 in the following dataset to b and c respectively:

dataset = Dataset[{
   <|"a" -> 1, "b" -> "x", "c" -> {1}|>,
   <|"a" -> 2, "b" -> "y", "c" -> {2, 3}|>,
   <|"a" -> 3, "b" -> "z", "c" -> {3}|>,
   <|"a" -> 4, "b" -> "x", "c" -> {4, 5}|>,
   <|"a" -> 5, "b" -> "y", "c" -> {5, 6, 7}|>,
   <|"a" -> 6, "b" -> "z", "c" -> {}|>}]
  1. What is the most efficient way to do it at scale, i.e. assuming the replacements are going to be across multiple/ all rows in large datasets.
  2. Is is possible to do it inline, i.e., in the current dataset, without having to recreate a new dataset?
  3. It would be preferrable if the updates are computed (i.e., not constants as shown in this toy example)
  • 1
    $\begingroup$ It's not really an answer but two things I would mention: 1) I've found Dataset to not be very handy for working with data. I tend to process data as associations and only create a Dataset at the end when I'm going to do queries / analysis. 2) Associations support structural sharing so #2 may not actually be a concern (see youtu.be/0ZnZlAKMZdg?t=843 ) -- obviously I don't know everything about what you're doing but if it's just about memory usage then it should be ok. $\endgroup$ – Dan Dec 10 '18 at 2:51

You can use ReplacePart directly on the dataset but it seems a tedious/inefficient way of doing this. It is scalable as long as you can use patterns or Alternative to select the positions you want to replace. I expect someone else will be able to provide a more efficient method that works directly on the dataset without first converting to a list of associations.

dataset=ReplacePart[dataset, {{3 | 4 | 5, "b"} -> b, {3 | 4 | 5, "c"} -> c}]

enter image description here

| improve this answer | |
  • $\begingroup$ This does not perform an in-place modification. If you evaluate dataset after the ReplacePart, it is unchanged. $\endgroup$ – Rohit Namjoshi Dec 10 '18 at 1:01
  • $\begingroup$ @RohitNamjoshi good point. I'll leave the answer up even though it does not provide in-place modification $\endgroup$ – Mike Honeychurch Dec 10 '18 at 5:21

I mentioned it in the comments but basically I like to keep it pretty simple and while I'm not sure if it is the fastest way to do it it is nice and clean and the memory usage is not as bad as it would seem due to Mathematica's good choice of persistent data structures to build Associations on.

In[1]:= MemoryInUse[] / 1000000 // N
Out[1]= 96.5472
In[2]:= data = ResourceData["US State Income"]
In[3]:= MemoryInUse[] / 1000000 // N
Out[3]= 117.604
(* Updating a field *)
In[4]:= data2 = Dataset[<|"State"->#State, "Income"->#Income/2.0|>&/@Normal@data]
In[5]:= MemoryInUse[] / 1000000 // N
Out[5]= 117.885
(* Appending a field *)
In[6]:= data3 = Dataset[<|#, "IncomeSquared"->#Income^2|>&/@Normal@data]
In[7]:= MemoryInUse[] / 1000000 // N
Out[7]= 118.574
| improve this answer | |

Actually, I don't now how to do that with a Dataset but with nested Lists and Association, in-place modification is possible:

assoc = Normal[dataset];
assoc[[3 ;; 5, {"b", "c"}]] = ConstantArray[{b, c}, 3];
assoc // Dataset

So, if you don't need any specific properties of Dataset, I'd suggest to stick with nested Associations.

| improve this answer | |
  • $\begingroup$ Im trying to manipualte a real-world dataset and am exploring using Dataset in MMA for its GroupBy and other convenience features like plotting by columns etc. $\endgroup$ – my account_ram Dec 9 '18 at 18:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.