When using Java objects through JLink, how can I multitask the objects?

Question

I use Mathematica 11.0.1 on Windows 10 and have a Java class which acts as a 'facade' for a headless browser (PhantomJS).

In Mathematica I instantiate the class through JLink as follows:

BROWSWER = JavaNew["org.nilostep.botautil.BrowserFacade"];

With getData I can scrape data off a website.

exp = JavaObjectToExpression[BROWSER@getData[url, css]];

So far so good.

Ideally, I would like to have a number of browsers working on the Internet at the same time ( to shorten the time it takes to download all data ).

To test this I instantiated two browser objects as follows:

BROWSWER1 = JavaNew["org.nilostep.botautil.BrowserFacade"];

BROWSWER2 = JavaNew["org.nilostep.botautil.BrowserFacade"];

In the Windows10 Task Manager I could clearly see two instances of PhantomJS.

Then in two different cells of a Mma Notebook I started:

exp1 = JavaObjectToExpression[BROWSER1@getData[url1, css1]];

exp2 = JavaObjectToExpression[BROWSER2@getData[url2, css2]];

So BROWSER1 and BROWSER2 are objects of the same Java class, running in one JLink session, but which both spawn a different browser process.

Both cells looked active, but in the Task Manager I could clearly see that first browser1 was active on CPU, and then browser2. It looks as though Mma waits for the browser to finish its work on the first cell, before it starts on the second one. - The computer has enough memory to handle several browser sessions.

Is there a way to let the browsers work simultaneously, 'multitask' the Java objects?

Garth Sheldon-Coulson says on http://clojuratica.weebly.com that "Mathematica is not designed for threads or concurrency. It has excellent support for parallel computation, but parallel evaluations are initiated from a single-threaded master kernel which blocks until all parallel evaluations return." Perhaps his Clojuratica is an option, or have other options been investigated.

( I will further investigate this from the Java side. )

Answer 1

Create a Java class implementing the Producer / Consumer design pattern. See also: Producer-Consumer problem. The class below is an example.

Then in Mathematica instantiate the class and submit all queries via the submitQuery method. Get one query result at a time via the getResult method.

package org.nilostep.botautil;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

/**
 * Created by nilo on 18/02/2017.
 */
public class WebRobot {
    BlockingQueue<List> wq = new ArrayBlockingQueue(100);
    BlockingQueue<List> rq = new ArrayBlockingQueue(100);

    public WebRobot() {
        setup(2);
    }

    public WebRobot(int n) {
        setup(n);
    }

    public void setup(int n) {
        ExecutorService executor = Executors.newFixedThreadPool(n);
        for (int i=0; i<n; i++) {
            executor.submit(new QueryWorker(i, wq, rq));
        }
    }

    public void submitQuery(String url, String css) throws InterruptedException {
        List<String> arl = new ArrayList();
        arl.add(url);
        arl.add(css);
        wq.put(arl);
    }

    public List<String> getResult() throws InterruptedException {
        return rq.take();
    }
}

The QueryWorker is a consumer of the query queue and a producer of the result queue.

package org.nilostep.botautil;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.BlockingQueue;

/**
 * Created by nilo on 18/02/2017.
 */
public class QueryWorker implements Runnable {

    private final int id;
    private final BlockingQueue<List> queryQueue;
    private final BlockingQueue<List> resultQueue;
    private final BrowserFacade browser = new BrowserFacade();

    QueryWorker(int id,
            BlockingQueue<List> q,
            BlockingQueue<List> r) {
        this.id = id;
        this.queryQueue = q;
        this.resultQueue = r;
    }

    public void run() {
        try {
            while (true) {
                List lst = queryQueue.take();
                String url = (String) lst.get(0);
                String css = (String) lst.get(1);
                resultQueue.put(browser.getData(url,css));
            }
        } catch (InterruptedException ex) {
            System.out.println(ex.toString());
        }
    }
}