Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I'm trying to parallelize some tasks which involve fetching data from a URL and processing it. This is the code:

keyId = 1;
SetSharedVariable[keyId];
Clear[keyLock];

EncodeURL[str_String] := 
 StringReplace[str, 
  x : Except[Alternatives @@ Characters@";/?:@=&$-_.+!*'()"] :> 
   ExternalService`EncodeString[x]]
ApiKeys =
   {
   "key1",
   ...
   ...
   "lastKey"
   };

Api[urlNoKey_] := 
 TimeConstrained[ApiInternal[urlNoKey], 5, Api[urlNoKey]]

ApiInternal[urlNoKey_] := 
 Block[{content, code, 
   url = EncodeURL[
     baseURL <> urlNoKey <> "?api_key=" <> ApiKeys[[keyId]]]},
  {content, code} = 
   Check[URLFetch[url, {"Content", "StatusCode"}], {$Failed, $Failed}];
  PrintTemporary[url];
  CriticalSection[{keyLock},
   If[keyId == Length@ApiKeys, keyId = 1, keyId++];
   ];
  Switch[code,
   200, ImportString[content, "JSON"],
   404, {},
   429 , LolApiInternal[urlNoKey], (* Rate time limit exceeded *)
   $Failed, Print["Api::error -> URLFetech returned $Failed"]; {},
   code, Print["Api::error -> Unkown code: ", code, 
    ".Content was: ", content]; {}
   ]
  ]

I am using a 4 core machine but since the computing is small and URLFetch is involved, I am using 8 kernels (using LaunchKernels[8])

I am trying to distribute the processing of a list of about 10.000 urls:

ParallelMap[Process@Api[#] &, urlList];

I get the following errors (and more) during the computation (usually, near the end)

http://i.imgur.com/PdC2ORK.png

Any idea why and how to solve it? Note that I am using FetchURL and ImportString (I think this last one uses a Java link)

share|improve this question
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.