# How to Set parts of indexed lists?

I would like to assign a list to an indexed variable and then change it using Part and Set like this:

matM[i] = ConstantArray[1, {10, 10}];
matM[i][[1,10]] = 10


Unfortunately I will get the error:

Set::setps: "matM[i] in the part assignment is not a symbol. "

Using Subscript will not work either:

Subscript[matM,i] = ConstantArray[1, {10, 10}];
Subscript[matM,i][[1,10]] = 10


will give the same error.

How can this be done?

-
Is using a list instead and doing matM[[1]][[1,10]]=10 an option for you application? Could you explain why you need an indexed variable? Then it would be easier to come up with a solution. (As you noticed, assigning to Part doens't work with DownValues...) –  Szabolcs Jun 22 '12 at 9:37
Szabolcs, as written below, I am a bit amazed at what seems 'illogical' in not being able to address downvalues. I might have to use the list-option as you suggest, but for a more symbolic reference like matM[a] it would be very nice to do it the way I suggested. I should be possible, shouldn't it? –  gwr Jun 22 '12 at 9:50
I had the impression that you were doing a numeric/programming type of work, not symbolic manipulation (since you were using Part to change arrays element by element). What exactly are you doing? –  Szabolcs Jun 22 '12 at 10:08
Just to make this clearer, I have edited the original question in order to make it clearer by having the index be a symbol. –  gwr Jun 22 '12 at 10:09
Szabolcs, I am just trying to have an indexed variable be a list or to be more precise an array. While in principle you are right that I might transfer every index to a numerical range and thus one could come around by simply adding the index-dimension to the arrray dimension, there is good reason to work with the 'sparse' definitions allowed by using an index for the variable itself. My problem at hand is that I am changing code already written. Adding the index in the variable name is a simple replacement. Adding it as an array dimension means a lot of work... –  gwr Jun 22 '12 at 10:14

I can offer a sort of a solution, which has its shortcomings, but will allow you (more or less) to use the syntax you want. The idea is that you mark symbols you need, as "references", and then execute your code in a special dynamic environment where Part is overloaded.

### Implementation

Here is the code. This function will mark you symbol as a reference.

ClearAll[makeReference];
makeReference[sym_Symbol] :=
sym /: Set[sym[index_], rhs_] :=
With[{index = index},
With[{heldVal = Hold[sym[index]] /. DownValues[sym]},
If[heldVal === Hold[sym[index]],
Module[{ref},
AppendTo[DownValues[sym],
HoldPattern[sym[index]] :> ref]
]
];
Hold[sym[index]] /. DownValues[sym] /. Hold[s_] :> (s = rhs);
]];


What happens is that we "softly" overload Set on this particular symbol via UpValues, so that an intermediate symbol is inserted where the actual data will be stored, and our symbol (for a given index) refers to that intermediate symbol. Since the latter has no restrictions on part assignments, we can assign parts of it directly at O(1) time.

However, the subtlety is that when we call Set[Part[s[ind],1,2],something], Set holds its first argument, and therefore, s can not communicate to Set that this is special (UpValues won't work here since the s is too deep inside an expression - on level 2 - while UpValues are only looked at at level 1). To solve this problem, we will overload Part, but do it locally within a local dynamic environment, to make this operation safer. This is a dynamic environment:

ClearAll[withModifiedPart];
SetAttributes[withModifiedPart, HoldAll];
withModifiedPart[code_] :=
InternalInheritedBlock[{Part},
Unprotect[Part];
Part /: Set[Part[sym_Symbol[index_], inds__], rhs_] :=
With[{index = index},
Hold[sym[index]] /. DownValues[sym] /.
Hold[s_] :> (s[[inds]] = rhs);
];
Protect[Part];
code];


### Tests

Now, we can test this:

ClearAll[a];
makeReference[a];


and then

withModifiedPart[
a[1] = Range[10];
a[1][[2]] = 100;
a[1]
]

 {1, 100, 3, 4, 5, 6, 7, 8, 9, 10}


Let's now measure some timings:

withModifiedPart[
a[1] = Range[100000];
Do[a[1][[i]] = a[1][[i]] + 1, {i, a[1]}];
a[1]
] // Short // AbsoluteTiming

  {1.5126953,{2,3,4,5,6,7,8,9,<<99984>>,99994,99995,99996,99997,
99998,99999,100000,100001}}


We can compare this to the time it takes for direct assignments:

aa = Range[100000];
Do[aa[[i]] = aa[[i]] + 1, {i, aa}]; // Short // AbsoluteTiming

 {0.2470703,Null}


So, for massive assignments, we are about 6 times slower (which I think is OK). We can also see how costly is the overloaded Part for normal assignments:

withModifiedPart[
aa=Range[100000];
Do[aa[[i]]=aa[[i]]+1,{i,aa}]
];//Short//AbsoluteTiming

  {0.2822266,Null}


from where it looks that those are slower by about 15 percents.

### Conclusions

The suggested solution requires 2 modifications to your code:

• call makeReference on symbols which you wish to use as indexed symbols with part assignment, prior to assigning to them.
• Execute all your code containing such assignments, inside the withModifiedPart environment.

So, your original code will be changed to

ClearAll[matM];
makeReference[matM];

withModifiedPart[
matM[i] = ConstantArray[1, {10, 10}];
matM[i][[1, 10]] = 10
]


What about safety? I would argue that, for this particular case, modifying Part is safe enough. This is because, first, Part is overloaded softly, via UpValues. This means that, when Part is not inside some head which holds arguments, it will execute normally before it would even "think" of a new definition. Now, when it is inside some head which holds arguments, the new definition will only work if that head is Set. Note that no ne rules were added to the Set itself. And since normally, assignments to indexed variables are not allowed anyway, we don't risk breaking some internal behavior.

The performance here is worse than for direct assignments to symbol's parts, but overall seems acceptable.

-
Leonid, what a solution, thank you. I had believed there is some 'easy' way - possibly upvalues - but your answer shows how deep the matter runs. –  gwr Jun 22 '12 at 13:47
@gwr Thanks :). In fact, since Part is overloaded via UpValues, and then only for Set, and part assignments for indexed symbols are normally prohibited anyway, I would even think that globally overloading Part could be ok (i.e. doing the same thing but without the dynamic invironment), in which case, you don't have to wrap your code in it. But, it is still better to be on the safe side, all it takes is just to wrap your top-level function(s) in withModifiedPart. Thanks for the accept as well. –  Leonid Shifrin Jun 22 '12 at 13:50
+1, nice one! What's the purpose of setting the DownValue by hand only when there's no other downvalue associated with the symbol, versus simply sym[index]=ref? –  Rojo Jun 22 '12 at 13:52
@Rojo Thanks:). As to the reason for my construct: because I already overload Set on this symbol, I can not use it (Set) to set the DownValue, since the new definition will be used, and we will enter an infinite recursion. –  Leonid Shifrin Jun 22 '12 at 13:54
Right, and furthermore you avoid creating a different symbol every time a certain index is reassigned a new value, leaking memory. My yet-non-DownValues-user instinct would have led me to tackle both issues differently. Using explicit DownValues is something that doesn't come straight to mind –  Rojo Jun 22 '12 at 14:07

Here is another way.

SetAttributes[set,HoldFirst]
set[symb_[idx__][[part__]],var_]:=Module[{x=symb[idx]},
x[[part]]=var;
symb[idx]=x
]

use[symb_]:=Module[{},
Unprotect[Part];
Set[symb[idx__][[part__]], var_] ^:= set[symb[idx][[part]],var];
Protect[Part];
]


testing with:

use[a]
a[x] = {1, 2, 3}
a[x][[2]] = 7


we get, as in @Mr.Wizard test:

{1, 7, 3}

I like the fact that this solution keep the Downvalues in a, instead of a random named Global variable x. I don't like to Unprotect Part, but in this case doesn't seem to be bad.

In time performance this is not a good answer when compared with @Leonid and @Mr.Wizard.

-

Unexpectedly I don't favor Leonid's approach. I feel it costs too much performance, and performance is a primary reason for using an assignment like this.

My method is to store the data in a direct assignment to a separate symbol and use a special function to effect the assignment.

SetAttributes[set, HoldFirst]

set[sym_ @ idx_, val_] :=
Module[{x},
Quiet[sym @ idx =.];

sym @ idx =. ^:= (sym /: sym @ idx =.; x =.);

set[sym[idx][[part__]], vv_] := x[[part]] = vv;

sym @ idx = x;
x = val
]


Use:

set[a[x], {1, 2, 3}];

set[a[x][[2]], 7];

a[x]

{1, 7, 3}


Care is taken by the function to remove old data when new is assigned. It can also be manually cleared with a[x] =. and the hidden symbol will also be cleared.

Leonid cautiously avoids global modification to Part or Set. One could take a more bold approach for convenience sake and probably be safe as the pattern is rather specific. (In this case it will only match for the exact symbol[index] that is initialized with set.) This could be done with:

set[sym_ @ idx_, val_] :=
Module[{x},
Quiet[sym @ idx =.];

sym @ idx =. ^:= (sym /: sym @ idx =.; x =.);

Unprotect[Part];
Set[sym[idx][[part__]], vv_] ^:= x[[part]] = vv;
Protect[Part];

sym @ idx = x;
x = val
]


One would still need the initial set[a[x], . . .] to set up, but assignments to parts would be done with a[x][[i]] = . . .

### Timings compared to Leonid's method

makeReference[a];

withModifiedPart[
a[1] = Range[100000];
Do[a[1][[i]] = a[1][[i]] + 1, {i, a[1]}];
]; // AbsoluteTiming

{0.6300009, Null}

AbsoluteTiming[
set[a[1], Range[100000]];
Do[set[a[1][[i]], a[1][[i]] + 1], {i, a[1]}];
]

{0.1800003, Null}

-
There is no question that by using a special syntax you can gain performance, I realized that from the start. The complexities of my solution are due to my attempt to keep the syntax as close to the original as possible. Performance is a valid concern, but not the one raised in the question, so I did not consider it a primary concern as long as the slowdown is not too dramatic (e.g. full order of magnitude or more). Anyway, nice code, and +1. –  Leonid Shifrin Jun 23 '12 at 12:08
As to performance, I became much more pragmatic about it recently, as I see more and more that the requirement to always stay within Mathematica is artificial. My current thinking is that Mathematica has several layers where it shines, such as 1. High-level (possibly rule-based) - very easy to put new ideas to code 2. Number crunching and other tasks where one can use Compile 3. Data manipulation, where one can use vectorized operations to make it reasonably fast (of course, I only talk about core proramming). But it also has gaps, in particular in the realm of mutable efficient ... –  Leonid Shifrin Jun 23 '12 at 12:26
... data structures and efficient general programming. These gaps can often be alleviated by using other languages (e.g. Java, but not only). I can foresee the proliferation of tools similar to my Java reloader, connecting Mathematica to other languages, so I value M mostly as a very interactive medium and development laboratory, a tool to create prototypes. Once I see that something is slow, I will localize the problem and use Java or C or some other language. We spend gigantic efforts on optimization while the code can often be written straightforwardly in other langs and save us time. –  Leonid Shifrin Jun 23 '12 at 12:31
@Leonid those are very valid points. I am biased because I tried to learn C many years ago and it only gave me headaches; then I discovered Mathematica and the joy of doing with one line of code what was taking dozens and I never looked back. Your Java Reloader makes using code copy and paste easy so it's hard to argue against building a Java library, though if I never learn Java I'll end up with "cargo cult programming." I am (largely thanks to you) slowly coming to see the value of using Mathematica as a framework for multiple languages. –  Mr.Wizard Jun 23 '12 at 17:32
@Leonid by the way, what is your take on the safety of the second option I present? It seems to me that overloading Part with a specific pattern is unlikely to cause problems. Do you agree? –  Mr.Wizard Jun 23 '12 at 17:37

Ok, I'll offer a solution that I think it's better suited to some of the use cases: use the parsing stage.

A simplistic implementation could be

ClearAll[makeReference];
makeReference[sym_] :=
MakeExpression[
RowBox[{ToString@Unevaluated@sym, "[", indexBox_, "]"}],
StandardForm] :=
MakeExpression[
ToString@Unevaluated@sym <> "\[LetterSpace]" <>
ToExpression[indexBox, StandardForm, ToString], StandardForm]


Now we don't need the dynamic environment any more, and we get faster results. Beware: this definition is not attached to the symbol, so to unset it one would need to ipmlement these as a fucntion that unsets the MakeExpression definition, and depending on your goal, maybe removing the intermediate symbols and copying the values straight to sym[index]. Just let know of further complications in trying to implement whatever you need.

Long PD

While looking at Leonid's idea, I thought that I wouldn't have done it that way, not mainly because I know better ones but because I tend to avoid explicit DownValues for whatever reason. What follows is just a reimplementation of @Leonid's answer avoiding the explicit use of DownValues (by basically saving the rules somewhere else). So, bear in mind, it's just an alternative implmeentation of Leonid's great code.

Since some non-experienced users may find his code a little cryptic and mine not, others may find mine a little cryptic for other reasons and his not. Hopefully this adds a few extra guys who end up understanding any of the ways to implement this. That's why I commented it quite a bit. Remember that in this version, just like Leonid's we do need the dynamic environment withModifiedPart that you can find in his answer.

According to a few tests, it is faster than Leonid's for writing new indexes and rewriting already indiced variables, but a little slower on reading and modification of parts of indexed variables, so I guess it depends on the use case

ClearAll[makeReference];
makeReference[sym_Symbol] :=
Module[{$guardEval = True,$guardOver = True},

(* This is where we'll be saving the intermediate symbol rules *)
sym["IntermediateSymbols"] = {};

(* This definition is the first one tried when setting a value on \
the indexed symbol. It replaces the sym[
index] by the intermediate symbol and tries setting again.
The next trial,
this definition won't be matched even if there wasn't any \
intermediate symbol defined *)
sym /: exp : Set[sym[_], _] /; $guardEval := Block[{$guardEval = False},
Unevaluated[exp] /. sym["IntermediateSymbols"]
];

(* This definition is only found if an intermediate symbol hasn't \
been defined yet. It assigns a new one,
appends it to the intermediate symbols list,
and sets the desider value *)
sym /: Set[sym[index_], rhs_] /; $guardOver := Block[{$guardOver = False},
Module[{ref},
sym[index] = ref;
AppendTo[sym["IntermediateSymbols"],
HoldPattern[sym[index]] :> ref];
ref = rhs
]
];

]

-
+1 Great job, thank you Rojo. I must admit as somewhat 'intermediate' user I certainly will need a bit of time to thoroughly understand the solutions offered by Leonid and you. –  gwr Jun 22 '12 at 15:53
Thanks! @grw, I just saw a tiny bug in the first solution (the one with MakeExpression) and fixed it. Be sure to use the new one. –  Rojo Jun 22 '12 at 15:57
+1, Interesting version. I thought about something similar, but did not want to introduce extra dependencies on extra global symbols. On my tests, your code is about 20 percent slower to modify the parts, than mine, however. I am on Win7 64 bit, M 8.0. –  Leonid Shifrin Jun 22 '12 at 17:43

Here is one way I found doing this:

setIndexed[var_, val_, index__] := Module[

{tempVar},

tempVar = var;
tempVar[[index]] = val;
tempVar
];

matM[i] = ConstantArray[1,{10,10}];

matM[i] = setIndexed[matM[i],10,1,10]


Seems to work fine. But maybe there is a more elegant way to do this?

-
As others have hinted, the more elegant way might be not to use an indexed array, but perhaps we need more information to know why an indexed array is the best solution? –  image_doctor Jun 22 '12 at 10:52
The reasons for using indexed variables as opposed to lists is addressed in the Mathematica reference (reference.wolfram.com/mathematica/tutorial/…). Why should these reasons - sparse definitions made only when needed - not apply when the variable contains an array? –  gwr Jun 22 '12 at 11:16
Clearly this is a good question, as it is generating a lot of debate :) Have you found the built in SparseArray doesn't quite fit your application? –  image_doctor Jun 22 '12 at 11:44
As I had written to Szabolcs the reason for choosing an index was the more simple way to change existing code. But the SparseArray is simply the list-solution with better allocation of resources in case of lots of empty cells? I can't see how this is a replacement for an indexed var which essentially allows dynamic programming? –  gwr Jun 22 '12 at 13:29
Testing this runs as fast as direct assignment for a non-indexed var. Does this solution have the same memory disadvantages as ReplacePart does? (the indexed-var needs to be copied and reassigned) –  gwr Jun 22 '12 at 14:53

matM[1] is a a call to the function matM using the argument 1, not an index. To assign the array to an index of matM, use:

 matM = {{}};
matM[[1]] = ConstantArray[1, {10, 10}];
matM[[1]][[1, 10]] = 10
matM[[1]]


The first line is just to make sure matM has a first element to write the array into.

Edit: ok so the exact semantics I chose may have been slightly to lighthearted, but the point is that matM[1] is definitely not a symbol, which is also what Mathematica complains about. Even so you can still assign a value to it, just like you would assign a "function" matM[1]=2, compared to matM[x_]=2. The difference being that the pattern is a constant 1 in one and and arbitrary match x_ in the second. But simply because you can assign a value to the pattern does not mean you can treat it like a Symbol, if you consider the example:

 matM[x_] = ConstantArray[1, {10, 10}];
matM[1][[1, 10]] = 10


It should be immediately apparent why Mathematica warns you that matM[1] is not a symbol. In your particular case, the "pattern" used for the rule definition just happens to be unique, which means that there is no real conceptual problem in redefining part of the contained matrix.

-
Saying that "it's a call to the function with the argument 1" is a bit misleading. It's quite common to use variables named x[1], x[2], x[3], ... for symbolic manipulation or other tasks when we don't know beforehand how many variables we'll be needed, or when there's a large number of variables that we need to work with and it's inconvenient to refer to them 'manually'. I think this is clear to the OP. –  Szabolcs Jun 22 '12 at 9:45
Thanks. But using the 'fixed' argument in a function is explicitly suggested in the Mathematica documentation (reference.wolfram.com/mathematica/tutorial/…) and for example in Sal Mangano's "Mathematica Cookbook". It is of course possible to make this an array adding the index dimension, but if - as in my real case - you want to use symbols like matM[a] is not so easy to do. –  gwr Jun 22 '12 at 9:47
@gwr I expanded my answer a bit, the important note is that matM[a] is not a symbol, and this was also what I was hinting towards originally. You can have some behavior which resembles what you have with symbols, but in other situations it falls short exactly because it isn't a symbol. –  jVincent Jun 22 '12 at 11:37
@jVincent, thank you for expanding your answer and it does help to clear the issue. Nevertheless it is a 'pity' that even the subscripted variable cannot be addressed by Part and Set. Essentially then you cannot have a truly indexed symbol? –  gwr Jun 22 '12 at 13:24
@gwr Indexed variables are really a hash-table under the name of the corresponding symbol, and it points at values rather than assignable memory locations, which is IMO quite natural. I think, my solution gives you the closest to "truly indexed symbols" you can get, since I am using intermediate symbols, which are, indeed, "truly indexed". –  Leonid Shifrin Jun 22 '12 at 13:35

No need for a custom function, you can use the built in ReplacePart:

matM[1] = ConstantArray[1, {10, 10}];

matM[1] = ReplacePart[matM[1], {2, 5} -> 7]


There are variations of ReplacePart that allow you change groups of elements.

-
The big problem with ReplacePart is that it copies entire list rather than modify a part in-place. What this means is that part assignment is now O(n) rather than O(1), where n is a number of elements in a list. And also, frequent list copying means an overhead due to excessive memory allocation, and possibly enhanced memory consumption (depending on how efficient is garbage collector / allocator). For all practical purposes, this is a disaster in most cases. –  Leonid Shifrin Jun 22 '12 at 11:22
@LeonidShifrin That all makes perfect sense, I'm not sure how large the OPs n` is. I guess you've traveled this route many times in the past ? –  image_doctor Jun 22 '12 at 11:49
Yes, I did :) –  Leonid Shifrin Jun 22 '12 at 12:02