# List damaged after update with Nothing [duplicate]

In version 10.4.1 (Win 7 Ent) I am implementing an algorithm that needs to update a list after each pass. I initially coded this with Nothing by removing items in the list that where no longer needed after each processing pass. This did not result in the expected outcome and I eventually sourced the error to the use of Nothing. A very minimal example follows.

SeedRandom[19384];
datN = RandomInteger[{1, 5}, 10]
(* {2, 1, 3, 5, 1, 2, 5, 2, 2, 2} *)


For this minimal example FirstPosition[5]@datN will represent the index of the item that needs to be removed after each processing pass. (I'm not trying to delete all 5's in the real code.) There are two 5's. I would expect the first in position 4 to be removed at the end of the first pass.

datN[[Echo@First@FirstPosition[5]@datN]] = Nothing;
datN
>> 4
(* {2, 1, 3, 1, 2, 5, 2, 2, 2} *)


Success! The first pass finds the first 5, removes it, and updates datN. Or does it? Lets try the second pass which should remove the 5 in position 6.

datN[[Echo@First@FirstPosition[5]@datN]] = Nothing;
datN
>> 6
(* {2, 1, 3, 1, 5, 2, 2, 2} *)


What has happened here? The correct position of 6 was identified but instead of removing the 5 there it removed the 2 in position 5. Let us try one more time to remove the 5 now in position 5.

datN[[Echo@First@FirstPosition[5]@datN]] = Nothing;
datN
>> 5
(* {2, 1, 3, 5, 2, 2, 2} *)


It has done it again. The correct position of 5 has been identified but the 1 in position 4 has been removed instead.

Now it gets interesting. On the fourth try and every subsequent try after the fourth it simply refuses to remove anything.

datN[[Echo@First@FirstPosition[5]@datN]] = Nothing;
datN
>> 4
(* {2, 1, 3, 5, 2, 2, 2} *)


The output above will repeat with the datN never being altered from the forth try onwards. Very odd.

Have I completely missed something or is this a bug? If so, is it still around in version 11?

Update

Clearly I have to live with it as it is. However, at this moment, I don't feel it is intuitively correct. I would understand the hold behavior if I used SetDelayed but having used Set it seems a reasonable expectation that the Nothings be removed. At least I understand its mechanism. I am dating myself a bit here but I was expecting a ReDim Preserve type of behavior. 10 Internet points to anyone that gets that reference without looking it up.

• Can confirm this buggy behavior using Mathematica v11.0.0 under Windows 10. – Karsten 7. Aug 15 '16 at 21:54
• @Karsten7. I will report to WRI. – Edmund Aug 15 '16 at 21:55
• Yeah, wow. Have a look at Trace[datN[[Echo@First@FirstPosition[5]@datN]] = Nothing, TraceInternal -> True] after doing all these steps. There is a {2, 1, 3, Nothing, Nothing, Nothing, 5, 2, 2, 2} in it. – Karsten 7. Aug 15 '16 at 22:03
• @Karsten7. Yes. But it clearly goes against the definition of Nothing in the documentation. Nothing should be removed from List. datN is a list. In addition, there are no Hold* attributes on datN. – Edmund Aug 15 '16 at 22:07
• I have marked this question as "already has an answer" because it is the same exact issue manifesting in a slightly different way. Leonid's answer already includes Nothing. This question will remain as a guidepost, but if desirable someone may wish to edit the original question to include Nothing as well. – Mr.Wizard Aug 15 '16 at 22:28

This is not a bug.

You are assuming that

arr = {1,2,3}
arr[[2]] = Nothing


removes the second element. It doesn't! It sets the second element to Nothing, just the same as if you used any other symbol.

What happens when you evaluate arr now is the following evaluation steps:

arr -> {1,Nothing,3}
{1,Nothing,3} -> {1,3}


However, the length of arr is still 3.

You can see this with

Definition[arr]
(* arr = {1, Nothing, 3} *)


If you want to remove that element, then you need the extra step

arr = arr


Update: I see the Edmund's comment was upvoted by several people, which shows that there's a widespread misunderstanding about what Nothing does. The documentation states that

Nothing is removed as part of the standard evaluation process.

That is, it is removed during evaluation when it appears explicitly as part of a list. It does not however interact in any way with Set or Part. arr[[2]] = Nothing does not behave in a special way. It just sets the second element of the array to Nothing.

Now when you evaluate arr, it goes through two evaluation steps, as I stated above, first into {1,Nothing,3} which then evaluates further into {1,3}. Thus the result of the evaluation of arr is a length-2 list, and Length[arr] returns 2. This does not mean that arr itself is a length-2 list. It only means that it evaluates to a length-2 list. You can see with OwnValues[arr] that it has 3 elements, the second being Nothing.

• I disagree. Evaluate Length@datN after each pass. It decreases. Also the documentation says that Nothing is removed from List. datN is a list. There are no Hold* Attributes on datN. The Nothings should be removed as is reasonably expected. – Edmund Aug 15 '16 at 22:04
• +1 - I recall an exact case like this (I think we both answered with the same explanation). Possible dupe? – ciao Aug 15 '16 at 22:05
• @Edmund Of course Length[arr] is 2 in my example because Length is not HoldAll! Yes, Nothing is removed from a list during evaluation. That is, after you evaluate arr, it changes to {1, Nothing, 3}, which then changes to {1,3}. However, Nothing doesn't interact with Set and Part in any way! The only special property is that it is removed during evaluation when it explicitly appears in a list. You understood the documentation. – Szabolcs Aug 15 '16 at 22:08
• @ciao Here is your duplicate. Wish I'd checked since it has a response by Leonid Shifrin that I am sure is much better than my own. – Daniel Lichtblau Aug 15 '16 at 22:20
• @DanielLichtblau Do you have an opinion on this? – Szabolcs Aug 16 '16 at 10:32

Citing the Documentation (emphasis is mine):

Nothing is removed as part of the standard evaluation process. It is not removed in expressions that are held or inactive.

So it should be removed upon evaluation, what actually happens, and there is no statement that it should be removed from the original unevaluated list.

• I would agree if I used SetDelayed. However, I used Set. – Edmund Aug 15 '16 at 22:17
• @Edmund You used Part which is very special in Mathematica because it allows in-place modification of a list without evaluation of the whole list! – Alexey Popkov Aug 15 '16 at 22:18

I don't think this is exactly a bug. But it is at minimum a woefully underdocumented feature, and identical to one that shows up when using Sequence[]. If you use Information you will get an idea of what is happening.

datN[[First@FirstPosition[5]@datN]] = Nothing;
datN

(* Out[891]= {2, 1, 3, 1, 2, 5, 2, 2, 2} *)

Information[datN]
(* GlobaldatN
datN={2,1,3,Nothing,1,2,5,2,2,2} *)


Now repeat:

datN[[First@FirstPosition[5]@datN]] = Nothing;
datN

(* Out[912]= {2, 1, 3, 1, 5, 2, 2, 2} *)
In[913]:= Information[datN]

(* GlobaldatN
datN={2,1,3,Nothing,1,Nothing,5,2,2,2} *)


The problem is as follows. First@FirstPosition[5]@datN evaluates datN, hence gets it's positioning "correct", that is, for the list with the Nothing removed. Part, when used as a left-hand-side of an expression (what we call "setpart"), is not actually evaluating datN. This is a mismatch.

I am not sure if there is a sound reason for this behavior of setpart. It might be necessitated by the HoldFirst attribute of Set. This behavior goes back to version 1 I believe, so I'd be quite reluctant to mess with it even if Set semantics might allow for that.

• Clearly I have to live with it as it is. However, at this moment, I don't feel it is intuitively correct. At least I understand its mechanism. – Edmund Aug 15 '16 at 22:20
• "I am not sure if there is a sound reason for this behavior of setpart." ← I think there is. I would find it quite disturbing if it didn't behave this way. To make it behave the way @Edmund thought it did would require evaluating the list in its entirety when only one element is being set. Now consider arr = {1,x,2}; x = 42; If arr[[3]] = Sequence[] triggered an evaluation of all of arr and re-setting it to the evaluation result, it would change it to arr = {1,42} instead of arr = {1,x}. That would be rather confusing. Also, what if evaluating x has side effects, e.g. ... – Szabolcs Aug 16 '16 at 6:54
• ... x := Print["foo"];? Even more confusing if x gets evaluated just because a different element of arr was changed! Of course in principle arr[[3]] = Sequence[] (or Nothing) could be made to remove the 3rd element without re-evaluating any other part of the list. But that would mean adding extra special properties to Sequence[]/Nothing that are unrelated to what they do at the moment. That would add additional complexity to and already complex language with too many surprising behaviours. At least the current behaviour is quite simple and clear once one understands it. – Szabolcs Aug 16 '16 at 6:57
• All good points, @Szabolcs. Though the current behavior has its anomalies. For example one could eval the list in intermediate steps in the current scheme, e.g. using side effects, and get inconsistencies. But overall I agree with you, the present semantics are the better way to go. – Daniel Lichtblau Aug 16 '16 at 14:15