Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I'm trying to parallelize some tasks which involve fetching data from a URL and processing it. This is the code:

keyId = 1;

EncodeURL[str_String] := 
  x : Except[Alternatives @@ Characters@";/?:@=&$-_.+!*'()"] :> 
ApiKeys =

Api[urlNoKey_] := 
 TimeConstrained[ApiInternal[urlNoKey], 5, Api[urlNoKey]]

ApiInternal[urlNoKey_] := 
 Block[{content, code, 
   url = EncodeURL[
     baseURL <> urlNoKey <> "?api_key=" <> ApiKeys[[keyId]]]},
  {content, code} = 
   Check[URLFetch[url, {"Content", "StatusCode"}], {$Failed, $Failed}];
   If[keyId == Length@ApiKeys, keyId = 1, keyId++];
   200, ImportString[content, "JSON"],
   404, {},
   429 , LolApiInternal[urlNoKey], (* Rate time limit exceeded *)
   $Failed, Print["Api::error -> URLFetech returned $Failed"]; {},
   code, Print["Api::error -> Unkown code: ", code, 
    ".Content was: ", content]; {}

I am using a 4 core machine but since the computing is small and URLFetch is involved, I am using 8 kernels (using LaunchKernels[8])

I am trying to distribute the processing of a list of about 10.000 urls:

ParallelMap[Process@Api[#] &, urlList];

I get the following errors (and more) during the computation (usually, near the end)


Any idea why and how to solve it? Note that I am using FetchURL and ImportString (I think this last one uses a Java link)

share|improve this question
add comment

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.