Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I'm working in some web scraping using Mathematica, and today, to speed-up the process, I used bash commands (xargs with parallel options with curl and wget) using ReadList or Run. I would like to do all this inside Mathematica, but I miss multi-thread capability with asynchronous evaluation. I could use more cores, but it's not the answer because the lag is due to time lag, not lack of processor capacity.

In bash I can do things like:

cat url.txt | xargs - n1 - P20 wget

Even with 4 cores it distribute the download process in 20 processes.

In Mathematica I can do this:

data = ParallelMap[Import,urlList]

But you get limited to cores number.

I thought that it would be natural to be multi-threaded if Mathematica is able to use multiple cores. Any clues?

share|improve this question
    
Did you try LaunchKernels[n]? –  belisarius Oct 3 '12 at 13:01
    
I think at some point one has to accept that Mathematica is not the best tool for every possible job. Here's an idea, though: why not take the source code of wget or aria2, patch it, and compile it as a MathLink executable or a "Wolfram Library" for use with LibraryLink? –  Oleksandr R. Oct 3 '12 at 14:11
    
Hi Belisairus. Great tip!.. I'll make some tests and put what I get. It appears to works. –  Murta Oct 3 '12 at 14:27
    
@OleksandrR. interesting idea. I'm wondering how nightmarish that would be ... Obviously, the functionality could be mapped to options/arguments, not simple, but it may be doable. –  rcollyer Oct 3 '12 at 14:31
    
@OleksandrR. seems easier to just call them via the command line from mathematica... –  acl Dec 8 '12 at 17:31
add comment

2 Answers 2

In Mathematica 9, you can now use URLFetchAsynchronous and URLSaveAsynchronous. These perform the HTTP request asynchronously in threads.

share|improve this answer
3  
And you can keep track of how many threads you are using with Length@AsynchronousTasks[] –  Gustavo Delfino Dec 8 '12 at 16:08
add comment

tks @belisarius! There is the improved code.

CloseKernels[]; LaunchKernels[16];
ParallelMap[Import,urlList]

My mistake was to think that the number of kernels was limited by the number of cores.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.