8
$\begingroup$

I'm working in some web scraping using Mathematica, and today, to speed-up the process, I used bash commands (xargs with parallel options with curl and wget) using ReadList or Run. I would like to do all this inside Mathematica, but I miss multi-thread capability with asynchronous evaluation. I could use more cores, but it's not the answer because the lag is due to time lag, not lack of processor capacity.

In bash I can do things like:

cat url.txt | xargs - n1 - P20 wget

Even with 4 cores it distribute the download process in 20 processes.

In Mathematica I can do this:

data = ParallelMap[Import,urlList]

But you get limited to cores number.

I thought that it would be natural to be multi-threaded if Mathematica is able to use multiple cores. Any clues?

|improve this question
$\endgroup$
  • $\begingroup$ Did you try LaunchKernels[n]? $\endgroup$ – Dr. belisarius Oct 3 '12 at 13:01
  • $\begingroup$ I think at some point one has to accept that Mathematica is not the best tool for every possible job. Here's an idea, though: why not take the source code of wget or aria2, patch it, and compile it as a MathLink executable or a "Wolfram Library" for use with LibraryLink? $\endgroup$ – Oleksandr R. Oct 3 '12 at 14:11
  • $\begingroup$ Hi Belisairus. Great tip!.. I'll make some tests and put what I get. It appears to works. $\endgroup$ – Murta Oct 3 '12 at 14:27
  • $\begingroup$ @OleksandrR. interesting idea. I'm wondering how nightmarish that would be ... Obviously, the functionality could be mapped to options/arguments, not simple, but it may be doable. $\endgroup$ – rcollyer Oct 3 '12 at 14:31
  • $\begingroup$ @OleksandrR. seems easier to just call them via the command line from mathematica... $\endgroup$ – acl Dec 8 '12 at 17:31
11
$\begingroup$

In Mathematica 9, you can now use URLFetchAsynchronous and URLSaveAsynchronous. These perform the HTTP request asynchronously in threads.

|improve this answer
$\endgroup$
  • 3
    $\begingroup$ And you can keep track of how many threads you are using with Length@AsynchronousTasks[] $\endgroup$ – Gustavo Delfino Dec 8 '12 at 16:08
6
$\begingroup$

tks @belisarius! There is the improved code.

CloseKernels[]; LaunchKernels[16];
ParallelMap[Import,urlList]

My mistake was to think that the number of kernels was limited by the number of cores.

|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.