I need to import some xml information from many pages. I have list of this pages. Doing it in single thread is too long. But I can't access multiple pages from the same IP at the same time. So I need to parallelize program execution with individual proxy settings for each kernel/thread. Is there any way to run several instances of Mathematica with individual proxy settings? It there any other tool to deal with such task?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I found a partial soulution by myself. When I used ParallelDo all kernels have the same settings. In Parallel Kernel Configuration I create a couple of new kernels. This are Master Kernels. Every one of them can have it's own proxy settings. It's possible to assigned different cells to different master kernels.
ExpressionCell[Defer[2 * 2], "Input", Evaluator -> "Local"]
I would like to know if there is a program way not only assign existing core but to create a new master core. Finally, I ended up using curl. WM created batch file, after it finished, WM processed the data. I did it so, because I can't handle with timeout option of URLFetch/URLRead. This option doesn't seem to work correctly.