# LibraryLink: What can we do with MInputStream and MOutputStream?

Version 9 introduced the WolframStreamsLibrary.h header for LibraryLink. It contains MInputStream and MOutputStream.

What can I do with this header? How can I use these functions? Are the documented anywhere?

Possibly related:

Update 2018 March: Partial spelunking results, someone else may want to pick up the trail.

Currently, my best guess is that this is for extending Mathematica with new streams. I believe it to be the analogue of DefineInputStreamMethod and DefineOutputStreamMethod. The struct MInputStream contains most of the functions that would be passed to DefineInputStreamMethod. The actual analogue is registerInputStreamMethod in WolframLibray.h, which I believe sets the equivalents of the ConstructorFunction and NameTestFunction. My (unverified!) guesses for the parameters of registerInputStreamMethod are:

• const char *name, same as name in DefineInputStreamMethod
• void (*ctor)(MInputStream, const char* msgHead, void* optionsIn), same as "ConstructorFunction". Should fill out the MInputStream struct passed to it. But in what format does it receive the options?
• mbool (*handlerTest)(void*, char*), probably the same as "NameTestFunction". But what is the void * parameter for? The methodData?
• void* methodData, possibly pointer to private data used by the stream implementation.
• void (*destroyMethod)(void* methodData), possibly for freeing the methodData? This is not the "CloseFunction".

There are still many key questions remaining before one could actually implement everything that is needed.

What is the methodData? A state unique to each instance of the stream, or shared by all of them? (I assume this is shared, and the individual state is in the MInputStream struct.) What exactly should each of these functions do?

An example implementation is woefully needed. After some more spelunking, I found one in SystemFiles/Links/SocketLink/Source/C. This example may provide enough information for us to write our own implementation, but I have not yet had the time/motivation to do all this. I still wanted to share my findings with others who may be interested in continuing the spelunking.

Note that the SocketLink example won't answer all questions: it often passes NULL pointers or simply ignores arguments (e.g the options in the constructor).

• I think these items are largely to do with the "data" (Hadoop, etc.) functionality being written by Taliesin Beynon and presumably to appear in version 10. (Other traces of this project are also visible already in version 9.) It wouldn't surprise me if they weren't documented until then, and as such, it may be that nobody can answer the question due to NDA. OTOH, maybe Joel Klein can comment at least on the stream functionality. – Oleksandr R. Sep 4 '13 at 9:33
• @OleksandrR. I was thinking they might also be related to the AsynchronousTasks functionality of v9, and the DataStore might provide a way to temporarily store the results before they're ready to be returned. – Szabolcs Sep 4 '13 at 14:41
• Szabolcs, what's up with the sudden large bounties? :) – Mr.Wizard Dec 10 '16 at 16:33
• @Mr.Wizard Don't know what to do with all my reputation ... Also, previously I didn't really believe that bounties would work (or that they would motivate people—they don't motivate me). But in the last week I got more than one unexpectedly good answer to questions I set a bounty on. If you know any worthy questions, I'm willing to "donate" reputation for bounties :) – Szabolcs Dec 10 '16 at 16:35
• @Szabolcs I see. I find bounties motivating only in that it indicates the question is particularly important to someone, and as from the beginning my biggest motivation in being here is the sense (hopefully not false) that I am helping people at least in some small way. Regarding "donation" I tried something like that myself once. I couldn't really tell if people appreciated that or not. If they did I am happy to "donate" more bounties myself. – Mr.Wizard Dec 10 '16 at 16:46