Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I've written a function to pull in the useful column of a large data file:

ImportNthColumn[fname_, n_] := With[{
  command = "!awk '{ print $" <> ToString[n] <> " }' "
 },
  Flatten@Import[command <> fname, "Table"]
]

and I'd like to apply this function to pull in data from every subdirectory, like so:

ImportNthColumnFromDirectories[dirs_, fname_, n_] := With[{
  targets = Map[# <> "/" <> fname&, dirs]
},
  Map[ImportNthColumn[#,n]&, targets]
]

When I use ImportNthColumnFromDirectories with a serial map, everything works perfectly. When I use ParallelMap, however, the data in the first and last elements of dirs import correctly but all other iterations return a zero-length list. What could cause this strange behavior?

share|improve this question
1  
Can you give a reproducible example? Without your data files it's hard to say much of anything. One thing you could do is use head on each of the files to make smaller versions that you could then upload. –  Taliesin Beynon Jul 30 at 21:09

1 Answer 1

Documentation say:

Import["!prog","format"] imports data from a pipe.

I suggest that ParallelMap sent many requests to external program in one time and it return errors, but Mathematica interpret errors as empty list.

PS: sorry for my english

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.