Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I'm trying to parallelize some tasks which involve fetching data from a URL and processing it. This is the code:

keyId = 1;
SetSharedVariable[keyId];
Clear[keyLock];

EncodeURL[str_String] := 
 StringReplace[str, 
  x : Except[Alternatives @@ Characters@";/?:@=&$-_.+!*'()"] :> 
   ExternalService`EncodeString[x]]
ApiKeys =
   {
   "key1",
   ...
   ...
   "lastKey"
   };

Api[urlNoKey_] := 
 TimeConstrained[ApiInternal[urlNoKey], 5, Api[urlNoKey]]

ApiInternal[urlNoKey_] := 
 Block[{content, code, 
   url = EncodeURL[
     baseURL <> urlNoKey <> "?api_key=" <> ApiKeys[[keyId]]]},
  {content, code} = 
   Check[URLFetch[url, {"Content", "StatusCode"}], {$Failed, $Failed}];
  PrintTemporary[url];
  CriticalSection[{keyLock},
   If[keyId == Length@ApiKeys, keyId = 1, keyId++];
   ];
  Switch[code,
   200, ImportString[content, "JSON"],
   404, {},
   429 , LolApiInternal[urlNoKey], (* Rate time limit exceeded *)
   $Failed, Print["Api::error -> URLFetech returned $Failed"]; {},
   code, Print["Api::error -> Unkown code: ", code, 
    ".Content was: ", content]; {}
   ]
  ]

I am using a 4 core machine but since the computing is small and URLFetch is involved, I am using 8 kernels (using LaunchKernels[8])

I am trying to distribute the processing of a list of about 10.000 urls:

ParallelMap[Process@Api[#] &, urlList];

I get the following errors (and more) during the computation (usually, near the end)

http://i.imgur.com/PdC2ORK.png

Any idea why and how to solve it? Note that I am using FetchURL and ImportString (I think this last one uses a Java link)

share|improve this question

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.