# MathLink C API: sending a List[] of so far unknown length

I would like to send a set of Mathematica expressions (i.e. not simple types that are easy to represent and temporarily store in C) in a List using the MathLink C API. I do not know how many there will be before I compute all of them, but I need their count to be able to call MLPutFunction.

How can I conveniently send a List without needing to know the number of elements beforehand?

One idea I had was to put them on a loopback link while counting them, then transfer the contents of the loopback link at once. But I was hoping for a simpler solution.

-
I have a solution for it, but I don't recall all of the details. Essentially, you start with an open List, and then each member is added as Sequence[dat,a] where a is the next data point, i.e. you make a nested list. Then you can close it by adding Sequence[] as your last point, collapsing the whole structure. Alternatively, you could use a loop-link. I'll write it up, if I get the chance. –  rcollyer Jan 31 '13 at 18:36
Instead of Sequence,which will lead to a quadratic complexity because of the run-time splicing, I would use some inert head and build a linked list, Flatten-ing this at the end. See my comments under Arnoud's answer. –  Leonid Shifrin Jan 31 '13 at 19:12
Could also use the InternalBag, as shown in Leonid's answer to the question on SO linked by Rolf? –  Oleksandr R. Jan 31 '13 at 19:25
@OleksandrR. But how will the bag help for transferring data via MathLink? –  Leonid Shifrin Jan 31 '13 at 19:27
@LeonidShifrin could send each element of the list one-at-a-time wrapped in StuffBag and then unpack them later on. Whether it's acceptably efficient to send each item individually would depend on the application, I suppose. –  Oleksandr R. Jan 31 '13 at 19:31
show 1 more comment

I consider the loopback link solution to this problem to be simple and ideal. I appreciate the cleverness of the LinkedList/Bag/Sequence types of solutions that have been proposed in other answers, but they seem to require a little too much thought for my taste.

This type of problem is one of the practical applications that motivated the creation of loopback links.

int err;
int count = 0;
while (test) {
....
MLPutInteger(loop, i);
count++;
}
MLClose(loop);

-
Does the loopback link interface provide a method for counting the elements already put on the link? –  rcollyer Feb 1 '13 at 15:04
@rcollyer But why would one have to need it? Such a counter is easy to set up, if necessary, during putting pieces onto the link, seems to me. –  Leonid Shifrin Feb 1 '13 at 15:32
@LeonidShifrin Either, I give each function I pass the link to access to a global variable (the counter), or I pass in the counter. In the first case, I break encapsulation. In the second, it increases the work I need to do to use it. By providing such a function, I can simplify how I use the link. –  rcollyer Feb 1 '13 at 16:48
@rcollyer Ok, good point. In addition to what you suggested, one could create a data structure incorporating the link and the counter (a struct), and pass that one. This will however require one to also define new functions for putting various types on the link, which would increment the counter, so this is still some work. –  Leonid Shifrin Feb 1 '13 at 16:59
@LeonidShifrin yes, you could define it that way. It is for things like this that c++ is superior: automatic type detection so that you can overload function declarations. From the user perspective, no more MLPutInteger, MLPutShortInteger, MLPutLongInteger, etc., only MLPut. The libs creators still need to make each one, or use a template, so not necessarily less work on their part. But, much simpler to use. I did a little work trying to set up such a c++ interface, but time constraints interfered. –  rcollyer Feb 1 '13 at 19:50

From that documentation page:

In order to call functions like MLPutFunction(), you need to know the length of the list you want to send. But by creating a sequence of nested Sequence objects, you can avoid having to know the length of your whole list in advance.

However, based on Leonid's comments below it is (much) more efficient to use an inert head like LinkedList so the example below is updated to use that.

This sets up the List around your result:

MLPutFunction(stdlink, "List", 1);
while( condition ) {
/* generate an element */


Create the next level LinkedList object.

 MLPutFunction(stdlink, "LinkedList", 2);


Put the element.

 MLPutInteger32(stdlink,  i );
}


This closes off your last LinkedList object.

MLPutFunction(stdlink, "LinkedList", 0);


Finally the kernel will need to flatten this expression:

Flatten[ result, Infinity, LinkedList ]

-
Sorry to bring the bad news, but splicing Sequence will make this a quadratic complexity solution, in the size of the list. It is unfortunate that this example made it into the docs. If the list is short and the complexity is not an issue, this is an acceptable solution, of course. –  Leonid Shifrin Jan 31 '13 at 19:05
It is easy to cure this solution by using some inert head instead of Sequence, say LinkedList, and make a linked list out of this. Then use Flatten[result, Infinity, LinkedList] at the end. This will have linear complexity. So, +1, assuming this modification to the answer. –  Leonid Shifrin Jan 31 '13 at 19:11
But isn't there also a limit of the order of 10^4 if I remember correctly? See also here –  Rolf Mertig Jan 31 '13 at 19:17
@LeonidShifrin why doesn't that exhibit the same complexity as Sequence? –  rcollyer Jan 31 '13 at 19:30
@rcollyer To understand this, you have to see why Sequence -based method has the quadratic complexity. Imagine that you have something like Sequence[1,Sequence[2,Sequence[3, Sequence[4]]]]. First, the inner Sequence gets spliced. We get Sequence[1,Sequence[2,Sequence[3,4]]]. And now, the evaluation starts over! But now, the inner Sequence has two elements, so they both will have to be copied during splicing. When you do this n times, each successive time you have to copy i elements. And basically, you copy an array, since Sequence` is array-based. –  Leonid Shifrin Jan 31 '13 at 19:40