Link server WSTP API feature not receiving incoming connections

Question

I'd like to apologize ahead of time that this may be a very long post but want to get all of this out there. Also, this post really only pertains to those with WSTP/MathLink experience.

My problem is that I have written (note: many parts are straight from the documentation) a working program that creates a link server (i.e. WSLinkServer type) and starts listening for connecting links. Once the server receives one, it should then return control flow to my program and then activate, write to, and finally close the link and shutdown the server. Using the Command Prompt netstat command, I can clearly see my program listening, but when I try to connect to the server from a Mathematica notebook (or kernel for that matter) to test the server, such as

link = LinkConnect["[email protected]", LinkProtocol->"TCPIP"];
If[LinkReadyQ[link], LinkRead[link]]

I then receive an error message and a $Failed return as seen here:

LinkConnect::linkc: Unable to connect to LinkObject[[email protected],2243,12].

Out[57]= $Failed

From my initial debugging, it seems that the C WSTP API function WSNewLinkServerWithPortAndInterface does not return when any link connection is attempted on the server from within Mathematica's notebook FE or a kernel. This API function is a blocking function by design (note: I've tried the async coded version too with the same result --- no connection), which waits for a connection and should return a WSLINK object when a link is received but it just keeps blocking and never returns. (note: the WSTP docs say words like "object" when referring to these API types but in the wstp.h file, you can see their definitions up to a point; so things like WSENV, WSLINK, WSLinkServer are nearly always some pointer to a struct or something similar).

Question(s):

(1) If my understanding is right that the server's listening link is akin to the so-called "main link", then do I need to create any additional links or loopback links for buffering that could be causing this problem? As a background, Mathematica's structure of having a main link, preemptive link, and service link corresponding to the variables $ParentLink, MathLink`$PreemptiveLink, and MathLink`$ServiceLink, where the main link is for the front end (my server program in this case) sending packets to the kernel (a connecting client in this case), the preemptive link is for the kernel sending results of requests back to the front end, and the service link is for the kernel making requests of the front end (opposite of main link).

(2) Is there another issue with the way I'm using the API functions here such as not correctly setting up the WSENV variable for say thread-safety, etc.? (Note: I've already tried this with the same result too. Be default, all WSLINK objects are not thread-safe unless specified prior to initializing the WSENV from which the links will be created).

(3) Am I compiling this program improperly? Could someone else try compiling it? (see below for my host info and compiler details)

(4) Is this just a bug in the WSTP API? It seems odd given the amount of fully documented C code for the link server feature and emphasis on the feature set found at Changes in WSTP for Interface 4.

Here is my C source code for a file called linkserver.c corresponding to the Makefile below (save them both in the same directory and ensure the Makefile variables are set correctly for your environment or compile manually):

#include "wstp.h"

void operate_link_server(WSENV env, unsigned short port, const char *interface)
{
        int error;
        WSLinkServer server = (WSLinkServer)0;
        WSLINK link = (WSLINK)0;

        server = WSNewLinkServerWithPortAndInterface(env, port, interface, NULL, &error);

        link = WSWaitForNewLinkFromLinkServer(server, &error);

        WSActivate(link);

        WSPutFunction(link, "Print", 1);
                WSPutString(link, "Hello client program.");
        WSEndPacket(link);
        WSFlush(link);

        WSClose(link);

        WSShutdownLinkServer(server);

        return;
}

int main(int argc, char **argv)
{
        WSEnvironmentParameter ep = (WSEnvironmentParameter)0;
        WSENV env = (WSENV)0;
        unsigned short port = 8000;
        const char *interface = "127.0.0.1";

        ep = WSNewParameters(WSREVISION, WSAPIREVISION);
        env = WSInitialize(ep);
        WSReleaseParameters(ep);

        operate_link_server(env, port, interface);

        WSDeinitialize(env);

        return 0;
}

Host and compiler info:

• I am running Mathematica v11.2 for Windows-x86-64 (Windows 10).
• I am using Mingw-w64 toolchain for the gcc compiler so I can compile from Cygwin 64-bit for Cygwin 64-bit or Windows 64-bit (I've also tried just gcc from Cygwin causing the cygwin1.dll dependency issue and have that built into my Makefile below as an option for people to try).

Here is the output of the command x86_64-w64-mingw32-gcc -v to get the compiler version info (can also do gcc -v if using the native compiler):

Using built-in specs.
COLLECT_GCC=x86_64-w64-mingw32-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-w64-mingw32/6.4.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
    Configured with: /cygdrive/i/szsz/tmpp/cygwin64/mingw64-x86_64/mingw64-x86_64-gcc-6.4.0-2.x86_64/src/gcc-6.4.0/configure --srcdir=/cygdrive/i/szsz/tmpp/cygwin64/mingw64-x86_64/mingw64-x86_64-gcc-6.4.0-2.x86_64/src/gcc-6.4.0 --prefix=/usr --exec-prefix=/usr --localstatedir=/var --sysconfdir=/etc --docdir=/usr/share/doc/mingw64-x86_64-gcc --htmldir=/usr/share/doc/mingw64-x86_64-gcc/html -C --build=x86_64-pc-cygwin --host=x86_64-pc-cygwin --target=x86_64-w64-mingw32 --without-libiconv-prefix --without-libintl-prefix --with-sysroot=/usr/x86_64-w64-mingw32/sys-root --with-build-sysroot=/usr/x86_64-w64-mingw32/sys-root --disable-multilib --disable-win32-registry --enable-languages=c,c++,fortran,lto,objc,obj-c++ --enable-fully-dynamic-string --enable-graphite --enable-libgomp --enable-libquadmath --enable-libquadmath-support --enable-libssp --enable-version-specific-runtime-libs --enable-libgomp --enable-libada --with-dwarf2 --with-gnu-ld --with-gnu-as --with-tune=generic --with-cloog-include=/usr/include/cloog-isl --with-system-zlib --enable-threads=posix --libexecdir=/usr/lib
    Thread model: posix
    gcc version 6.4.0 (GCC)

Here is my Makefile for compiling this program which people can change a few settings/variables to compile using a different compiler if needed. Note the directories where wsprep.exe, wstp.h, and wstp64i4.dll need to (or can) be located. Change as needed as I am using this from Cygwin 64-bit with directories ~/bin contains wsprep.exe, ~/lib contains wstp64i4.dll, ~/devel/include containing wstp.h, and ~/devel/lib containing wstp64i4.lib and any other .lib files generated using the dlltool as needed depending on your platform.

# DESCRIPTION:
# This makefile can be used to build one or more programs using Cygwin
# and the GNU Compiler Collection. Specifically, it is a cross setup to 
# build the program on an x86_64-pc-cygwin machine that will be run on a 
# host x86_64-pc-cygwin machine but with the compiler setup to produce
# code for an x86_64-w64-mingw32 (the [Mingw-w64][2] project which allows     
# GCC to compile on a Windows system and in my case run without dependency
# on the cygwin1.dll so we can run this program in Windows by develop it in
# Cygwin on Windows---if this is confusing read [here][2]). The CC variable
# to gcc, the DLLTOOL to dlltool, and the IMPORTLIB_TYPE variable to cyg if
# you want to compile for cygwin to create a so-called 'native' build where
# build, host, and target are all the same, creating a program dependent on
# the cygwin1.dll that will run on Windows as long as the cygwin1.dll is 
# visible to the program---something not required with the below settings.

# USAGE:
# To build all programs, use the command 'make all'.  To build the
# linkserver program, use the command 'make linkserver'.  Ensure this 
# file is called 'Makefile' and in the same directory as the source code
# file 'linkserver.c'.

# NOTE:
# Ensure that any appropriate read, write, and/or execute permissions are
# set for any of the directories and/or binaries handled by this Makefile.

# Portions of this makefile require the use of GNU make.
# see http://www.gnu.org/software/make for more information.

# Compiler variables.
CC = x86_64-w64-mingw32-gcc
DLLTOOL = x86_64-w64-mingw32-dlltool
#EXTRA_CFLAGS = -mwindows -mwin32
#EXTRA_CFLAGS = -mconsole -mwin32
#EXTRA_CFLAGS = -mwin32

# Compiler additions/resources directory variables.
CADDSDIR = $(HOME)/devel
INCDIR = $(CADDSDIR)/include
LIBDIR = $(CADDSDIR)/lib

# Other binaries/preprocessor executables directory variables.
BINDIR = $(HOME)/bin

# System additions/resources directory variables.
# This directory should contain the export/shared/runtime library .dll file and the
# directory should be included in the PATH environment variable during program
# execution time.
EXPORTLIBDIR = $(HOME)/lib

# WSTP developer kit export/shared/runtime library variables.
EXPORTLIB_PREFIX = wstp
EXPORTLIB_ARCH = 64
EXPORTLIB_INTERFACE_NUMBER = 4
EXPORTLIB_NAME = $(EXPORTLIB_PREFIX)$(EXPORTLIB_ARCH)i$(EXPORTLIB_INTERFACE_NUMBER)
EXPORTLIB = $(EXPORTLIB_NAME).dll

# WSTP developer kit import libary file name variable (e.g. a .lib file).
# Can be removed for default, s for static, m for Microsoft, or cyg for Cygwin.
IMPORTLIB_TYPE = 
IMPORTLIB_NAME = $(EXPORTLIB_NAME)$(IMPORTLIB_TYPE)
IMPORTLIB = $(IMPORTLIB_NAME).lib

# WSTP developer kit .tm file preprocessor tool executable name variable (e.g. 'wsprep.exe$
WSPREP = $(BINDIR)/wsprep.exe
# TODO: Add support for building up this command from a make command.
# Extra command line flags and arguments for 'wsprep.exe'.
#WSPREP_EXTRA_CFLAGS = -lines

# Command used to clean/remove files from the build directory.
RM = rm

# Array of all of the names of the binaries to build using the command 'make all'.
BINARIES = linkserver

all : $(BINARIES)    

linkserver : linkserver.o $(LIBDIR)/$(IMPORTLIB)
        $(CC) $(EXTRA_CFLAGS) -I$(INCDIR) linkserver.o -L$(LIBDIR) -l$(IMPORTLIB_NAME) -o $

%.o : %.c
        $(CC) -c $(EXTRA_CFLAGS) -I$(INCDIR) $<

%tm.c : %.tm
        $(WSPREP) $? -o $@

%cyg.lib :
        gendef $(EXPORTLIBDIR)/$(EXPORTLIB) #TODO: Add support for any flags.
        $(DLLTOOL) --as-flags=--64 -m i386:x86-64 -k --input-def $(EXPORTLIB_NAME).def --o$
        @ $(RM) $(EXPORTLIB_NAME).def

clean :
        @ $(RM) -rf *.o *tm.c $(BINARIES)

Note: This question only pertains to Interface 4 as the Link Server features were added with that release.

Answer 1

You are missing just one bit of secret sauce to get this to work. When you connect to a LinkServer link directly using WSTP/MathLink functionality, you need to use "-linkoptions 4" in the arguments to open the link. The value 4 is MLUseUUIDTCPIPConnection in wstp.h. In WL, this becomes

link = LinkConnect["[email protected]", LinkProtocol->"TCPIP", LinkOptions -> 4];

With this one change, your program will work.

I learned this only by looking at the kernel source code, to see how it uses the LinkServer feature internally. I don't know if this crucial information is documented anywhere. I think that perhaps that detail is handled automatically if you use the higher-level functions for automatic discovery of LinkServers, like WSResolveLinkService(). I don't know how to use that set of functions, however.

If you were connecting to your program from C or Java, you would use an array of link arguments like {"-linkname", "8000", "-linkhost", "127.0.0.1", "-linkprotocol", "tcpip", "-linkoptions", "4"}.

Your program is derived from the sample code in the WSTP documentation, but I think it probably isn't all that useful for you. That sample program shows you how to start a LinkServer and listen for client connections, but what do you do with the link returned from WSWaitForNewLinkFromLinkServer()? You probably want a kernel, or even a pool of kernels, at this end of the connection, so that clients can send WL computations. You could launch a kernel from your C program, and shuttle expressions back and forth between your link to the kernel and links to clients that have attached to your LinkServer. You could even write a modest amount of code to handle a pool of kernels, so that each client could get its own kernel. But if you just want a single long-lived kernel that can be attached to, and shared by, multiple clients, then you can write the whole thing in WL, with no C code required. This is possible because there are some obscure functions in WL for creating and managing LinkServers.

Let's take a little trip into "you-gotta-be-kidding-me-how-is-anyone-outside-Wolfram-supposed-to-know-that?" land. Put the following code into a file, say, LinkServerCode.wl:

server = WSTP`LinkServer`LinkServerCreate["[email protected]"];

While[$KeepRunning =!= False,
    links = WSTP`LinkServer`GetLinks[server];
    If[Length[links] > 0,
        Print["Client connecting"];
        MathLink`AddSharingLink[#, MathLink`AllowPreemptive->True, MathLink`ImmediateStart->True]& /@ links;
    ];
    Pause[0.1]
]

WSTP`LinkServer`LinkServerClose[server]

I'll comment more on that code in a minute.

Now you can launch the Wolfram Engine from the command line, telling it to run the code in that file:

wolframscript -file /Users/tgayley/Documents/LinkServerCode.wl

This will launch the Wolfram Engine and stay running (because of the While loop). In just a few lines of WL code, we have created a kernel that will run forever, waiting for any number of clients to attach to it and perform computations. The clients all use the same address to connect ("[email protected]"), so there is no hassle about having to create a different link name for each client. There is just one kernel, so of course all the clients will see the same kernel state, meaning that if one client sets x=42, then other clients will see that value of x.

The kernel will exit if any client evaluates $KeepRunning=False.

The code that runs the server is straightforward, just obscure because it uses undocumented functions. WSTP`LinkServer`GetLinks[] returns the list of links to newly-attaching clients. It is like WSWaitForNewLinkFromLinkServer(), except that it doesn't block. MathLink`AddSharingLink[] is what tells the kernel to pay attention to input arriving on the link. You can think of it as the multiple-link analogue of setting $ParentLink. The kernel maintains a set of links on which it waits for input to arrive, and internally sets $ParentLink to each one in turn as needed. The AllowPreemptive->True option allows the kernel to be receptive to input arriving on the link even while the kernel is evaluating something else (this is how you can drag a slider, or look up documentation pages, in the notebook front end while a cell is evaluating). The While[] loop never finishes, so without AllowPreemptive->True, the client link would never be serviced. ImmediateStart->True is needed for a similar reason. Without ImmediateStart, the kernel doesn't begin paying attention to the link until the evaluation that called AddSharingLink[] finishes, but as noted above, this is a While loop that never finishes.

Answer 2

This is unfortunately not a solution to the main problem yet, but I spent way to much time to not share the information. I tried several different ways and finally gave up. I created a public repository with my example that uses CMake and can be built on any machine out of the box.

• MathematicaLinkServer

The only thing (except from having a usable CMake installation) that needs to be adjusted is the line 7 in CMakeLists.txt. I guess on Windows you might be able to comment this out and your Mathematica installation will be found automatically.

To answer your question (3) first: Yes, I believe you have compiled it correctly. I get the exact same behavior and I used the CMake built-setup in quite a few projects.

Question (1) is a bit out of scope. At the moment, we try to establish one simple WSLink connection without concerning that the front end will set-up several links to a kernel. However, if you are interested in how the kernel sets up the other links, I strongly recommend looking at the sources of the java LinkSnooper. There are quite some comments in there and you see how they intercept this process to monitor all links. You can find the sources by evaluating

FileNames["LinkSnooper.java", {$InstallationDirectory}, Infinity]

Unfortunately, question (2) is just not answerable by someone not working for Wolfram. The only thing we have is the header file and the few c-code examples in the documentation. It's embarrassing that Wolfram introduces such features without a detailed tutorial that gives a complete example. All I can say is that it sounds like the WSLinkServer should be doing what you suspect.

If you look at my code, you'll see that opposed to your example, I set up the link server with a NULL interface which means that I let it decide the port and the hostname. The reason is that one of my first suspicions was that somehow the explicit specification of host and port did not work.

Instead, I'm creating a new server and print it's host and port so that can be used directly in Mathematica

unsigned short port = WSPortFromLinkServer(server, &error);
const char *interface = WSInterfaceFromLinkServer(server, &error);
fprintf(stderr, "link = LinkConnect[\"%d\", LinkProtocol -> \"TCPIP\", LinkHost -> \"%s\"]", port, interface);
WSWaitForNewLinkFromLinkServer(server, &error);

The second difference is that I registered a call-back function that should be called when a new link is established. The reason for changing this was that I debugged your program and set a break-point. As you have found out yourself, the WSWaitForNewLinkFromLinkServer never returns and I thought I tried a different way. However, my talkToMe function is never called either.

So I tried without having any server running

which is the exact behavior we see when we evaluate it with a server. My conclusion is that we somehow misunderstood how all this is supposed to work. I will report this to Wolfram and ask if they can provide further information and possibly a small example.