I use Mathematica 11.0.1 on Windows 10 and have a Java class which acts as a 'facade' for a headless browser (PhantomJS).
In Mathematica I instantiate the class through JLink as follows:
BROWSWER = JavaNew["org.nilostep.botautil.BrowserFacade"];
With getData I can scrape data off a website.
exp = JavaObjectToExpression[BROWSER@getData[url, css]];
So far so good.
Ideally, I would like to have a number of browsers working on the Internet at the same time ( to shorten the time it takes to download all data ).
To test this I instantiated two browser objects as follows:
BROWSWER1 = JavaNew["org.nilostep.botautil.BrowserFacade"]; BROWSWER2 = JavaNew["org.nilostep.botautil.BrowserFacade"];
In the Windows10 Task Manager I could clearly see two instances of PhantomJS.
Then in two different cells of a Mma Notebook I started:
exp1 = JavaObjectToExpression[BROWSER1@getData[url1, css1]]; exp2 = JavaObjectToExpression[BROWSER2@getData[url2, css2]];
So BROWSER1 and BROWSER2 are objects of the same Java class, running in one JLink session, but which both spawn a different browser process.
Both cells looked active, but in the Task Manager I could clearly see that first browser1 was active on CPU, and then browser2. It looks as though Mma waits for the browser to finish its work on the first cell, before it starts on the second one. - The computer has enough memory to handle several browser sessions.
Is there a way to let the browsers work simultaneously, 'multitask' the Java objects?
Garth Sheldon-Coulson says on http://clojuratica.weebly.com that "Mathematica is not designed for threads or concurrency. It has excellent support for parallel computation, but parallel evaluations are initiated from a single-threaded master kernel which blocks until all parallel evaluations return." Perhaps his Clojuratica is an option, or have other options been investigated.
( I will further investigate this from the Java side. )