I'm working in some web scraping using Mathematica, and today, to speed-up the process, I used bash commands (xargs with parallel options with curl and wget) using ReadList
or Run
.
I would like to do all this inside Mathematica, but I miss multi-thread capability with asynchronous evaluation. I could use more cores, but it's not the answer because the lag is due to time lag, not lack of processor capacity.
In bash I can do things like:
cat url.txt | xargs - n1 - P20 wget
Even with 4 cores it distribute the download process in 20 processes.
In Mathematica I can do this:
data = ParallelMap[Import,urlList]
But you get limited to cores number.
I thought that it would be natural to be multi-threaded if Mathematica is able to use multiple cores. Any clues?
wget
oraria2
, patch it, and compile it as a MathLink executable or a "Wolfram Library" for use with LibraryLink? $\endgroup$ – Oleksandr R. Oct 3 '12 at 14:11