Mathematica has a flexible streams interface. Functions like Read can read from a "stream" that may be backed by a file, pipe, string, network resource, etc. We can even create our own custom streams.
I am looking to implement an importer for a certain file format in C. How can I make it work with arbitrary Mathematica streams?
More generally, how can I create a C++ iostreams-interoperable interface that reads from / writes to Mathematica streams?
LibraryLink has a WolframStreamsLibrary.h header, as well as some related functions in WolframLibrary.h. What is the purpose of these? Can I use them to read from arbitrary streams, or are they designed to implement new Mathematica stream types in C? My impression is that they're intended for the latter.
An alternative idea is to try to use Mathematica callbacks to fill the C++ input stream's buffer:
- Use either library callback functions or MathLink
- Identify the stream by an integer, the second argument of
InputStream, e.g.Cases[Streams[], InputStream[_, 15]] - Use
ReadList[stream, Character, n]to readncharacters, and use them to fill the C++ input stream buffer. Unfortunately, this is very slow, but I am not sure how to read only up to a certain amount of data from a stream that may not haveBinaryFormat -> True(otherwiseBinaryReadListwould work). (Related: Why does ReadString complain about the stream not being opened in binary format?)
A big problem with this approach is that it is only practical if the C program is the exclusive user of the stream, or if it will read it all the way to the end. With this approach, the C code might read more data than it needs, and the stream might not be rewindable.
I am looking for advice on how to approach this problem, and feedback on the specific implementation ideas above.
