# Web scraping and multi-threading in Mathematica

I'm working in some web scraping using Mathematica, and today, to speed-up the process, I used bash commands (xargs with parallel options with curl and wget) using ReadList or Run. I would like to do all this inside Mathematica, but I miss multi-thread capability with asynchronous evaluation. I could use more cores, but it's not the answer because the lag is due to time lag, not lack of processor capacity.

In bash I can do things like:

cat url.txt | xargs - n1 - P20 wget


Even with 4 cores it distribute the download process in 20 processes.

In Mathematica I can do this:

data = ParallelMap[Import,urlList]


But you get limited to cores number.

I thought that it would be natural to be multi-threaded if Mathematica is able to use multiple cores. Any clues?

-
Did you try LaunchKernels[n]? –  belisarius Oct 3 '12 at 13:01
I think at some point one has to accept that Mathematica is not the best tool for every possible job. Here's an idea, though: why not take the source code of wget or aria2, patch it, and compile it as a MathLink executable or a "Wolfram Library" for use with LibraryLink? –  Oleksandr R. Oct 3 '12 at 14:11
Hi Belisairus. Great tip!.. I'll make some tests and put what I get. It appears to works. –  Murta Oct 3 '12 at 14:27
@OleksandrR. interesting idea. I'm wondering how nightmarish that would be ... Obviously, the functionality could be mapped to options/arguments, not simple, but it may be doable. –  rcollyer Oct 3 '12 at 14:31
@OleksandrR. seems easier to just call them via the command line from mathematica... –  acl Dec 8 '12 at 17:31

In Mathematica 9, you can now use URLFetchAsynchronous and URLSaveAsynchronous. These perform the HTTP request asynchronously in threads.

-
And you can keep track of how many threads you are using with Length@AsynchronousTasks[] –  Gustavo Delfino Dec 8 '12 at 16:08
CloseKernels[]; LaunchKernels[16];