I've written a function to pull in the useful column of a large data file:
ImportNthColumn[fname_, n_] := With[{
command = "!awk '{ print $" <> ToString[n] <> " }' "
},
Flatten@Import[command <> fname, "Table"]
]
and I'd like to apply this function to pull in data from every subdirectory, like so:
ImportNthColumnFromDirectories[dirs_, fname_, n_] := With[{
targets = Map[# <> "/" <> fname&, dirs]
},
Map[ImportNthColumn[#,n]&, targets]
]
When I use ImportNthColumnFromDirectories
with a serial map, everything works perfectly. When I use ParallelMap
, however, the data in the first and last elements of dirs
import correctly but all other iterations return a zero-length list. What could cause this strange behavior?