Is it possible to have an expression x such that its elements are only evaluated when they are individually accessed, for example when evaluating statements like x[[1]] or f/@x? The basic idea is that x is a list of Get commands, each of which loads a large expression (gigabytes), so that x can be terabytes in size yet it remains fairly manipulable within Mathematica.

What I thought would work was something like:

x={Unevaluated[Get[...]],Unevaluated[Get[...]],...}


The problem with that however is that Unevaluated doesn't get stripped when I need it to, i.e. when calling something like f[x[[1]]], because of Mathematica's evaluator semantics. The only workable alternative is to use Hold instead of Unevaluated, but that requires that I manually call ReleaseHold every time, which is ugly. I was hoping for something entirely transparent.

-
The answer is yes. Basically,something similar is done in this answer. It can be combined with this great answer on lazy lists, to make the lazy nature of data loading more explicit, but this may not be necessary. To rephrase, the first linked answer currently lacks generic stream interface and centralized memory manager for streams - both can be written rather easily on top of it though. –  Leonid Shifrin Jul 2 '13 at 22:17
The other limitation of the first linked answer is that it only deals with specific expressions - lists with large sub-lists. Again, this can be generalized to arbitrary expressions and their parts, and I actually plan to do so, but this has not been done yet. –  Leonid Shifrin Jul 2 '13 at 22:19
Not sure whether to consider this one a duplicate of the one on file-backed lists, let's see what others think. –  Leonid Shifrin Jul 2 '13 at 22:21
Thanks @LeonidShifrin for the pointers. Am I wrong in concluding that the solutions you mentioned require that I overload Part and related functions, and thus are not general in the sense that I have to know a priori which functions will operate on the expression? If that is the case then it doesn't help me, as the purpose of my question was to have a solution that would work transparently. I may have misunderstood your solutions however. –  Mohammed AlQuraishi Jul 2 '13 at 22:42
Yes, you will have to know the functions. Or, support all core functions in Mathematica, which is doable but a lot of work. Basically, their is no magic spell here - for example, in order to fully support packed arrays when they were introduced, a huge amount of work was required to overload all important functions in Mathematica, on this (then new) data structure. A lazy stream is no different - since it is not yet offered as a part of the language, one would need to define those functions one want to work on it, oneself. In Java, one would have to implement an interface, etc. –  Leonid Shifrin Jul 2 '13 at 22:47