Custom iterable objects

Question

I understand that to create objects with custom behavior with respect to existing code one uses upvalues. However, I'm at loss when it comes to guess what interface is expected from objects fed to built-in functions.

As a practical example, I would like to create a custom object DiskList that doesn't store its values in memory but dynamically reads them from a file. It is easy enough to write an upvalue for it so that the expression mydisklist[[3]] works as expected, but how do I make such object able to be passed to Map?

Answer 1

Let's say that DiskList is just a symbolic wrapper that can trigger the reading of a file, with the semantic that words are the "atomic" elements. We might consider two different semantics for how Part and Map might work.

For demonstration purposes, let's say we have the text of the U.S. constitution, ExampleData[{"Text", "USConstitution"}], stored in a file with path pathToFile.

compute the structure that Map/Part can operate on natively

DiskList /: Part[DiskList[file_File], index_Integer] := Part[ReadList[file, Word], index];
DiskList /: Map[f_, DiskList[file_File]] := Map[f, ReadList[file, Word]];

DiskList[File[pathToFile]][[5]]
(* "of" *)

DiskList[File[pathToFile]][[10000]]
(* warning that part 10000 doesn't exist *)

StringLength /@ DiskList[File[pathToFile]] // Short
(* {8, 2, 3, 6, 2, 3, 6, 7, 2, 5, 2, 4, 1, 4, 7, 6, 9, 8, 6, <<8044>>, 3, 3, 8, 2, 3, 8, 3, 16, 5, 4, 7, 5, 2, 8, 2, 15, 5, 4, 11} *)

StringJoin[#, "TAG"] & /@ DiskList[File[pathToFile]] // Short
(* {"PREAMBLETAG", "WeTAG", "theTAG", <<8076>>, "shallTAG", "haveTAG", "intervened.TAG"} *)

One nice thing about this strategy is that we get all of the enhanced behaviors for free:

DiskList /: Part[DiskList[file_File], args___] := Part[ReadList[file, Word], args]
DiskList /: Map[f_, DiskList[file_File], args___] := Map[f, ReadList[file, Word], args];

This allows us to do stuff like this:

DiskList[File[pathToFile]][[5 ;; 7]]
(* {"of", "the", "United"} *)

DiskList[File[pathToFile]][[10000]]
(* EndOfFile *)

force the iterative semantics

DiskList /: Part[DiskList[file_File], index_Integer] :=
  Module[
    {stream = OpenRead[file], value},
    Skip[stream, Word, index - 1];
    value = Read[stream, Word];
    Close[stream];
    value];
DiskList /: Map[f_, DiskList[file_File]] :=
  Module[
    {stream = OpenRead[file], value = {}, next},
    next = Read[stream, Word];
    While[
      EndOfFile =!= next,
      AppendTo[value, f[next]];
      next = Read[stream, Word]];
   Close[stream];
   value];

The advantage here is the ability to short circuit. This might be useful for Part, but not particularly for Map.

This strategy might be more useful for Scan if our goal is to perform some operation on each value read from the file rather than produce an output.

We have the immediate problem, however, of how to support the full behavior that Part/Map/Scan normally support. How should we handle level specs in Map? Do we want to build out our own support for Span?

comments

Regarding

what interface is expected from objects fed to built-in functions

this is one of those situations where different languages operate under different computational semantics, and it's generally unsafe (or just tedious and unnecessary) to try to impose one language's semantics on another. Mathematica has no notions of interface and object in the sense that OO languages do. As was pointed out in the comments, to Mathematica, everything is an expression, and expressions are almost synonymous to syntax trees.

Specifically, Map is not a generic (as in parameterized types) operator that defers certain responsibilities to the types/objects passed to it. Map simply implements computational behaviors on the expressions passed to it. So you can't "hook" into it by providing overriding behaviors in your custom "object classes".

Since the rules in UpValues are applied before those in DownValues, you can think of up values as interrupting the "typical" sequence of evaluation. You're simply telling the evaluation engine to consult the rules for a symbol in the body of the expression before consulting those of the head of the expression. That's it. You're not injecting a bundle of special, overriding implementations (an object) into a function template.