10
$\begingroup$

I'm working in some web scraping using Mathematica, and today, to speed-up the process, I used bash commands (xargs with parallel options with curl and wget) using ReadList or Run. I would like to do all this inside Mathematica, but I miss multi-thread capability with asynchronous evaluation. I could use more cores, but it's not the answer because the lag is due to time lag, not lack of processor capacity.

In bash I can do things like:

cat url.txt | xargs - n1 - P20 wget

Even with 4 cores it distribute the download process in 20 processes.

In Mathematica I can do this:

data = ParallelMap[Import,urlList]

But you get limited to cores number.

I thought that it would be natural to be multi-threaded if Mathematica is able to use multiple cores. Any clues?

$\endgroup$
5
  • $\begingroup$ Did you try LaunchKernels[n]? $\endgroup$ Oct 3, 2012 at 13:01
  • $\begingroup$ I think at some point one has to accept that Mathematica is not the best tool for every possible job. Here's an idea, though: why not take the source code of wget or aria2, patch it, and compile it as a MathLink executable or a "Wolfram Library" for use with LibraryLink? $\endgroup$ Oct 3, 2012 at 14:11
  • $\begingroup$ Hi Belisairus. Great tip!.. I'll make some tests and put what I get. It appears to works. $\endgroup$
    – Murta
    Oct 3, 2012 at 14:27
  • $\begingroup$ @OleksandrR. interesting idea. I'm wondering how nightmarish that would be ... Obviously, the functionality could be mapped to options/arguments, not simple, but it may be doable. $\endgroup$
    – rcollyer
    Oct 3, 2012 at 14:31
  • $\begingroup$ @OleksandrR. seems easier to just call them via the command line from mathematica... $\endgroup$
    – acl
    Dec 8, 2012 at 17:31

3 Answers 3

12
$\begingroup$

In Mathematica 9, you can now use URLFetchAsynchronous and URLSaveAsynchronous. These perform the HTTP request asynchronously in threads.

$\endgroup$
1
  • 4
    $\begingroup$ And you can keep track of how many threads you are using with Length@AsynchronousTasks[] $\endgroup$ Dec 8, 2012 at 16:08
6
$\begingroup$

tks @belisarius! There is the improved code.

CloseKernels[]; LaunchKernels[16];
ParallelMap[Import,urlList]

My mistake was to think that the number of kernels was limited by the number of cores.

$\endgroup$
0
$\begingroup$

As of version 11 see URLSubmit and URLDownloadSubmit

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.