I've written a function to pull in the useful column of a large data file:

ImportNthColumn[fname_, n_] := With[{
  command = "!awk '{ print $" <> ToString[n] <> " }' "
  Flatten@Import[command <> fname, "Table"]

and I'd like to apply this function to pull in data from every subdirectory, like so:

ImportNthColumnFromDirectories[dirs_, fname_, n_] := With[{
  targets = Map[# <> "/" <> fname&, dirs]
  Map[ImportNthColumn[#,n]&, targets]

When I use ImportNthColumnFromDirectories with a serial map, everything works perfectly. When I use ParallelMap, however, the data in the first and last elements of dirs import correctly but all other iterations return a zero-length list. What could cause this strange behavior?

  • 1
    $\begingroup$ Can you give a reproducible example? Without your data files it's hard to say much of anything. One thing you could do is use head on each of the files to make smaller versions that you could then upload. $\endgroup$ – Taliesin Beynon Jul 30 '14 at 21:09

Documentation say:

Import["!prog","format"] imports data from a pipe.

I suggest that ParallelMap sent many requests to external program in one time and it return errors, but Mathematica interpret errors as empty list.

PS: sorry for my english


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.