11
$\begingroup$

The context:

I am planning to visualize some results from my MMA code using D3.js. So I wanted to make a simple REST api using the light weight SparkJava framework. Given my fairly nonexistent knowledge of Java, I needed something simple. I would like to make a Spark based api that will be called by the web visualization frontend in our local university network. The visualization data will be coming from MMA.

Following this simple and well written Spark tutorial, I am using IntelliJ IDEA and here goes my pom.xml.

Building on the IDE works but how to get a JAR with dependencies?

This can be achieved by adding JLink to Library or ClassPath via Project Setting in IntelliJ. Without that I get the error.

enter image description here

After adding JLink through the IDE com.wolfram.JLink.* is resolved and I can compile my java class. Launching the application from IntelliJ works fine.

enter image description here enter image description here

Running the java class from the IDE gets the jetty server started up under my localhost at port 4567 and I can now go to the specified routers of Spark Java.

  1. http://localhost:4567/fib/15
  2. http://localhost:4567/name?age=12

To call the router /name, I am loading a MMA function that returns a little chunk of dummy JSON. In the above Java code this file test_json.m is loaded as the following.

 for (int i = 0; i<size; i++) {
            try {
                kl.add(MathLinkFactory.createKernelLink("-linkmode launch -linkname 'math -mathlink'"));
                (kl.get(i)).discardAnswer();
                (kl.get(i)).evaluate("<<src/main/resources/mma_package/test_json.m");
                (kl.get(i)).discardAnswer();
            } catch (MathLinkException e) {
                System.out.println("Fatal error opening link: " + e.getMessage());
                if ((kl.get(i)) != null)
                    (kl.get(i)).close();
                return;
            }
        }

The main problem

I could not find a way to build a executable JAR via pom.xml configuration. Going the mvn install:install way (Apache Guide) for 3rd party local repository does not solve the problem. The heartbreaking failure to locate JLinkNativeLibrary persists as command-line build with mvn also fails. Though similar to the problem here, I guess it is not a Java 8 issue here, as build and run from IDE, works just fine even though my Linux is running oracle java-8.

How can we configure the pom.xml so that we can produce a fat jar with all dependencies resolved and which can be then deployed on a 64bit Linux (say Ubuntu 14.04) system using the traditional command like.

java -jar hellodocker_jar.jar

I am also fine if we can get the JAR running by referencing java classpath as suggested in here by Todd Gayley.

java -cp "hellodocker_jar.jar:/usr/local/Wolfram/Mathematica/10.3/SystemFiles/Links/JLink/JLink.jar" SparkMMA.Hello

This too leads to the following error about not finding the JLinkNativeLibrary!

Fatal error: cannot find the required native library named JLinkNativeLibrary.
Exception in thread "main" java.lang.UnsatisfiedLinkError: com.wolfram.jlink.NativeLink.MLOpenString(Ljava/lang/String;[Ljava/lang/String;)J
    at com.wolfram.jlink.NativeLink.MLOpenString(Native Method)
    at com.wolfram.jlink.NativeLink.<init>(NativeLink.java:117)
    at com.wolfram.jlink.MathLinkFactory.createMathLink0(MathLinkFactory.java:281)
    at com.wolfram.jlink.MathLinkFactory.createMathLink(MathLinkFactory.java:194)
    at com.wolfram.jlink.MathLinkFactory.createKernelLink0(MathLinkFactory.java:171)
    at com.wolfram.jlink.MathLinkFactory.createKernelLink(MathLinkFactory.java:65)
    at SparkMMA.Hello.main(Hello.java:29)

Handling multiple KernelLink connections from Java in a robust and stable way?

In the above code example I am opening four KernelLink and giving them jobs one after another. This connections remain open as long as the Spark server is running. Any suggestions on best practices when considerable number of requests hit the rest api end points? May be one can think of a ingenious method to close/refresh old links and restart new ones in a clever manner. But one needs to be careful about what happens to the requests that comes at the api end when such link restart/refresh is triggered via another thread simultaneously.

It will be great to test how far we can push KernelLink via HTTP benchmarking tools like wrk. After running the above code from IDE and starting the server, I used wrk to test the performance.

wrk -c 64 -d 30s http://localhost:4567/fib/15

Running 30s test @ http://localhost:4567/fib/15
  2 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    20.39ms   23.12ms 263.47ms   86.07%
    Req/Sec     2.16k   391.08     2.93k    70.64%
  128413 requests in 30.03s, 26.70MB read
Requests/sec:   4276.83
Transfer/sec:      0.89MB

Immediately after that I hit the other router

wrk -c 64 -d 30s http://localhost:4567/name?age=120

Running 30s test @ http://localhost:4567/name?age=120
  2 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   297.57ms  277.07ms   1.84s    76.74%
    Req/Sec   126.34     35.60   232.00     66.95%
  7483 requests in 30.02s, 3.40MB read
Requests/sec:    249.27
Transfer/sec:    115.89KB

There are few answers in the site about similar sort of JLink issue but none helped me to solve this problem the maven way. Hence I thought of writing a detailed question here.

I feel the Java experts here can shade some light and we can together push the wrk benchmark to a nice new Christmas or New Year record.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.