How to Set parts of indexed lists?

Question

I would like to assign a list to an indexed variable and then change it using Part and Set like this:

matM[i] = ConstantArray[1, {10, 10}];
matM[i][[1,10]] = 10

Unfortunately I will get the error:

Set::setps: "matM[i] in the part assignment is not a symbol. "

Using Subscript will not work either:

Subscript[matM,i] = ConstantArray[1, {10, 10}];
Subscript[matM,i][[1,10]] = 10

will give the same error.

How can this be done?

Answer 1

I can offer a sort of a solution, which has its shortcomings, but will allow you (more or less) to use the syntax you want. The idea is that you mark symbols you need, as "references", and then execute your code in a special dynamic environment where Part is overloaded.

Implementation

Here is the code. This function will mark you symbol as a reference.

ClearAll[makeReference];
makeReference[sym_Symbol] :=
  sym /: Set[sym[index_], rhs_] :=
     With[{index = index},
      With[{heldVal = Hold[sym[index]] /. DownValues[sym]},
        If[heldVal === Hold[sym[index]],
           Module[{ref},
              AppendTo[DownValues[sym], 
                    HoldPattern[sym[index]] :> ref]
           ]
        ];
        Hold[sym[index]] /. DownValues[sym] /. Hold[s_] :> (s = rhs);
      ]];

What happens is that we "softly" overload Set on this particular symbol via UpValues, so that an intermediate symbol is inserted where the actual data will be stored, and our symbol (for a given index) refers to that intermediate symbol. Since the latter has no restrictions on part assignments, we can assign parts of it directly at O(1) time.

However, the subtlety is that when we call Set[Part[s[ind],1,2],something], Set holds its first argument, and therefore, s can not communicate to Set that this is special (UpValues won't work here since the s is too deep inside an expression - on level 2 - while UpValues are only looked at at level 1). To solve this problem, we will overload Part, but do it locally within a local dynamic environment, to make this operation safer. This is a dynamic environment:

ClearAll[withModifiedPart];
SetAttributes[withModifiedPart, HoldAll];
withModifiedPart[code_] :=
   Internal`InheritedBlock[{Part},
      Unprotect[Part];
      Part /: Set[Part[sym_Symbol[index_], inds__], rhs_] :=
        With[{index = index},
          Hold[sym[index]] /. DownValues[sym] /. 
               Hold[s_] :> (s[[inds]] = rhs);
        ];
      Protect[Part];
      code];

Tests

Now, we can test this:

ClearAll[a];
makeReference[a];

and then

withModifiedPart[
   a[1] = Range[10];
   a[1][[2]] = 100;
   a[1]
]

 {1, 100, 3, 4, 5, 6, 7, 8, 9, 10}

Let's now measure some timings:

withModifiedPart[
    a[1] = Range[100000];
    Do[a[1][[i]] = a[1][[i]] + 1, {i, a[1]}];
    a[1]
] // Short // AbsoluteTiming

  {1.5126953,{2,3,4,5,6,7,8,9,<<99984>>,99994,99995,99996,99997,
    99998,99999,100000,100001}}

We can compare this to the time it takes for direct assignments:

aa = Range[100000];
Do[aa[[i]] = aa[[i]] + 1, {i, aa}]; // Short // AbsoluteTiming

 {0.2470703,Null}

So, for massive assignments, we are about 6 times slower (which I think is OK). We can also see how costly is the overloaded Part for normal assignments:

withModifiedPart[
  aa=Range[100000];
  Do[aa[[i]]=aa[[i]]+1,{i,aa}]
 ];//Short//AbsoluteTiming

  {0.2822266,Null}

from where it looks that those are slower by about 15 percents.

Conclusions

The suggested solution requires 2 modifications to your code:

• call makeReference on symbols which you wish to use as indexed symbols with part assignment, prior to assigning to them.
• Execute all your code containing such assignments, inside the withModifiedPart environment.

So, your original code will be changed to

ClearAll[matM];
makeReference[matM];

withModifiedPart[
  matM[i] = ConstantArray[1, {10, 10}];
  matM[i][[1, 10]] = 10
]

What about safety? I would argue that, for this particular case, modifying Part is safe enough. This is because, first, Part is overloaded softly, via UpValues. This means that, when Part is not inside some head which holds arguments, it will execute normally before it would even "think" of a new definition. Now, when it is inside some head which holds arguments, the new definition will only work if that head is Set. Note that no ne rules were added to the Set itself. And since normally, assignments to indexed variables are not allowed anyway, we don't risk breaking some internal behavior.

The performance here is worse than for direct assignments to symbol's parts, but overall seems acceptable.

Answer 2

Unexpectedly I don't favor Leonid's approach. I feel it costs too much performance, and performance is a primary reason for using an assignment like this.

My method is to store the data in a direct assignment to a separate symbol and use a special function to effect the assignment.

SetAttributes[set, HoldFirst]

set[sym_ @ idx_, val_] :=
 Module[{x},
   Quiet[sym @ idx =.];

   sym @ idx =. ^:= (sym /: sym @ idx =.; x =.);

   set[sym[idx][[part__]], vv_] := x[[part]] = vv;

   sym @ idx = x;
   x = val
 ]

Use:

set[a[x], {1, 2, 3}];

set[a[x][[2]], 7];

a[x]

{1, 7, 3}

Care is taken by the function to remove old data when new is assigned. It can also be manually cleared with a[x] =. and the hidden symbol will also be cleared.

Leonid cautiously avoids global modification to Part or Set. One could take a more bold approach for convenience sake and probably be safe as the pattern is rather specific. (In this case it will only match for the exact symbol[index] that is initialized with set.) This could be done with:

set[sym_ @ idx_, val_] :=
 Module[{x},
   Quiet[sym @ idx =.];

   sym @ idx =. ^:= (sym /: sym @ idx =.; x =.);

   Unprotect[Part];
   Set[sym[idx][[part__]], vv_] ^:= x[[part]] = vv;
   Protect[Part];

   sym @ idx = x;
   x = val
 ]

One would still need the initial set[a[x], . . .] to set up, but assignments to parts would be done with a[x][[i]] = . . .

Timings compared to Leonid's method

makeReference[a];

withModifiedPart[
  a[1] = Range[100000];
  Do[a[1][[i]] = a[1][[i]] + 1, {i, a[1]}];
]; // AbsoluteTiming

{0.6300009, Null}

AbsoluteTiming[
  set[a[1], Range[100000]];
  Do[set[a[1][[i]], a[1][[i]] + 1], {i, a[1]}];
]

{0.1800003, Null}

Answer 3

This is not a precise answer to the actual question, but the list of answers is from 2012 and IMO does urgently lack a note about Association. Since version 10 (2014) these will solve the underlying problem by offering an alternative for such cases:

ClearAll@matM
matM = Association[]
matM[i] = ConstantArray[1, {10, 10}];
matM[[Key[i], 1, 10]] = 10

For every case where one uses downvalues as a hashtable using Association instead like shown will also have many other advantages. I have not made tests but I would be very surprised if the above wouldn't be much more efficient than any of the alternatives. Of course using an Association will not work in all cases, e.g. when one needs a mixture of patterns and constant "indices" in the downvalues of a symbol. But for a vast majority of appications where one runs into the mentioned problem nowadays Association will certainly be the best solution.

Answer 4

Ok, I'll offer a solution that I think it's better suited to some of the use cases: use the parsing stage.

A simplistic implementation could be

ClearAll[makeReference];
makeReference[sym_] := 
 MakeExpression[
   RowBox[{ToString@Unevaluated@sym, "[", indexBox_, "]"}], 
   StandardForm] := 
  MakeExpression[
   ToString@Unevaluated@sym <> "\[LetterSpace]" <> 
    ToExpression[indexBox, StandardForm, ToString], StandardForm]

Now we don't need the dynamic environment any more, and we get faster results. Beware: this definition is not attached to the symbol, so to unset it one would need to implement these as a function that unsets the MakeExpression definition, and depending on your goal, maybe removing the intermediate symbols and copying the values straight to sym[index]. Just let know of further complications in trying to implement whatever you need.

Long PD

While looking at Leonid's idea, I thought that I wouldn't have done it that way, not mainly because I know better ones but because I tend to avoid explicit DownValues for whatever reason. What follows is just a reimplementation of @Leonid's answer avoiding the explicit use of DownValues (by basically saving the rules somewhere else). So, bear in mind, it's just an alternative implmeentation of Leonid's great code.

Since some non-experienced users may find his code a little cryptic and mine not, others may find mine a little cryptic for other reasons and his not. Hopefully this adds a few extra guys who end up understanding any of the ways to implement this. That's why I commented it quite a bit. Remember that in this version, just like Leonid's we do need the dynamic environment withModifiedPart that you can find in his answer.

According to a few tests, it is faster than Leonid's for writing new indexes and rewriting already indiced variables, but a little slower on reading and modification of parts of indexed variables, so I guess it depends on the use case

ClearAll[makeReference];
makeReference[sym_Symbol] :=
 Module[{$guardEval = True, $guardOver = True},
  
  (* This is where we'll be saving the intermediate symbol rules *)
  sym["IntermediateSymbols"] = {};
  
  (* This definition is the first one tried when setting a value on \
the indexed symbol. It replaces the sym[
  index] by the intermediate symbol and tries setting again. 
  The next trial, 
  this definition won't be matched even if there wasn't any \
intermediate symbol defined *)
  sym /: exp : Set[sym[_], _] /; $guardEval := 
   Block[{$guardEval = False},
    Unevaluated[exp] /. sym["IntermediateSymbols"]
    ];
  
  (* This definition is only found if an intermediate symbol hasn't \
been defined yet. It assigns a new one, 
  appends it to the intermediate symbols list, 
  and sets the desider value *)
  sym /: Set[sym[index_], rhs_] /; $guardOver := 
   Block[{$guardOver = False},
    Module[{ref},
     sym[index] = ref;
     AppendTo[sym["IntermediateSymbols"], 
      HoldPattern[sym[index]] :> ref];
     ref = rhs
     ]
    ];
  
  ]

Answer 5

No need for a custom function, you can use the built in ReplacePart:

matM[1] = ConstantArray[1, {10, 10}];

matM[1] = ReplacePart[matM[1], {2, 5} -> 7]

There are variations of ReplacePart that allow you change groups of elements.

Answer 6

Here is one way I found doing this:

setIndexed[var_, val_, index__] := Module[

  {tempVar},

  tempVar = var;
  tempVar[[index]] = val;
  tempVar
];

matM[i] = ConstantArray[1,{10,10}];

matM[i] = setIndexed[matM[i],10,1,10]

Seems to work fine. But maybe there is a more elegant way to do this?

Answer 7

matM[1] is a a call to the function matM using the argument 1, not an index. To assign the array to an index of matM, use:

 matM = {{}};
 matM[[1]] = ConstantArray[1, {10, 10}];
 matM[[1]][[1, 10]] = 10
 matM[[1]]

The first line is just to make sure matM has a first element to write the array into.

Edit: ok so the exact semantics I chose may have been slightly to lighthearted, but the point is that matM[1] is definitely not a symbol, which is also what Mathematica complains about. Even so you can still assign a value to it, just like you would assign a "function" matM[1]=2, compared to matM[x_]=2. The difference being that the pattern is a constant 1 in one and and arbitrary match x_ in the second. But simply because you can assign a value to the pattern does not mean you can treat it like a Symbol, if you consider the example:

 matM[x_] = ConstantArray[1, {10, 10}];     
 matM[1][[1, 10]] = 10

It should be immediately apparent why Mathematica warns you that matM[1] is not a symbol. In your particular case, the "pattern" used for the rule definition just happens to be unique, which means that there is no real conceptual problem in redefining part of the contained matrix.

Answer 8

Here is another way.

SetAttributes[set,HoldFirst]
set[symb_[idx__][[part__]],var_]:=Module[{x=symb[idx]},
    x[[part]]=var;
    symb[idx]=x
]

use[symb_]:=Module[{},
    Unprotect[Part];
    Set[symb[idx__][[part__]], var_] ^:= set[symb[idx][[part]],var];
    Protect[Part];
]

testing with:

use[a]
a[x] = {1, 2, 3}
a[x][[2]] = 7

we get, as in @Mr.Wizard test:

{1, 7, 3}

I like the fact that this solution keep the Downvalues in a, instead of a random named Global variable x. I don't like to Unprotect Part, but in this case doesn't seem to be bad.

In time performance this is not a good answer when compared with @Leonid and @Mr.Wizard.