1
$\begingroup$

I need to import some xml information from many pages. I have list of this pages. Doing it in single thread is too long. But I can't access multiple pages from the same IP at the same time. So I need to parallelize program execution with individual proxy settings for each kernel/thread. Is there any way to run several instances of Mathematica with individual proxy settings? It there any other tool to deal with such task?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I found a partial soulution by myself. When I used ParallelDo all kernels have the same settings. In Parallel Kernel Configuration I create a couple of new kernels. This are Master Kernels. Every one of them can have it's own proxy settings. It's possible to assigned different cells to different master kernels.

ExpressionCell[Defer[2 * 2], "Input", Evaluator -> "Local"]

I would like to know if there is a program way not only assign existing core but to create a new master core. Finally, I ended up using curl. WM created batch file, after it finished, WM processed the data. I did it so, because I can't handle with timeout option of URLFetch/URLRead. This option doesn't seem to work correctly.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.