I thought I had an idea that might lead to an answer to @yohbs' question, but as I was trying to check out the usage of MemoryConstrained[expr,b,failexpr], I discovered behavior I did not expect.
The following aborts immediately; neither k nor t are set. (Please don't worry about the Rule -- I added "Abort"-> merely to distinguish an aborted computation):
Clear[t,k];
MemoryConstrained[Table[k = j; t[j] = Range[10j], {j, 1000}],
2199, "Aborted" -> Table[t[j], {j,k}]]
Out[2]= Aborted->Table[t[j],{j,k}]
I guess that it takes more than 2199 bytes to build the Table expression and it does not even evaluate one iteration. If the limit is raised one byte to 2200, then the command does not abort at all.
Clear[t,k];
MemoryConstrained[Table[k = j; t[j] = Range[10j], {j,1000}],
2200, "Aborted" -> Table[t[j], {j, k}]] // Short
Out[4]= //Short= {{1,2,3,4,5,6,7,8,9,10},<<998>>,{1,2,<<9997>>,10000}}
As you can see below, 41M were used to store the result, not to mention the other 41M stored in t[_].
ByteCount[%]
Out[5]= 41277832
The last list, Range[10000], is stored in t[1000] and exceeds the limit of 2200 bytes:
ByteCount[t[1000]]
Out[6]= 80144
Any Range[n] with n equal to 256 or more causes an abort:
MemoryConstrained[Range[256], 2200]
Out[7]= $Aborted
ByteCount[Range[256]]
Out[8]= 2160
ByteCount[Range[257]]
Out[9]= 2416
(It is believable that there are more than 40 bytes of overhead in computing Range and that the chunking of memory causes the 256 byte jump, although these are just a random stabs at an explanation.)
Does someone know how to make MemoryConstrained abort when the limit is 2200? Or can someone explain when memory request counts toward the MemoryContrained limit?
Addendum
Thanks for the help so far. I'm on Mac OS 10.8.2, Mma 9.0.0, slightly old MacBook Pro (~2008), 6GB RAM.
I restarted Mathematica and tried again. Same result. However, changing the limit on the iterator j from 1000 to 100 got it work. Strange.
Clear[t,k];
MemoryConstrained[Table[k = j; t[j] = Range[10 j], {j, 100}],
2200, "Aborted" -> Table[t[j], {j, k}]] // Short
Aborted->{{1,2,3,4,5,6,7,8,9,10},{1,<<18>>,20},t[3]}

MemoryConstrainedbefore version 8 was released. "For version 8 the memory allocator has been rewritten and improved. (What a small sentence for such a huge endeavor and such a fine execution.)" (Oliver Ruebenkoenig) – Alexey Popkov Jan 1 at 20:15Tableauto compiling for the 1000? What happens if you play withSetSystemOptions["CompileOptions" -> {"TableCompileLength" -> 0}]? – ruebenko Jan 2 at 18:41