I'm trying to launching my first jobs on LightWeight Grid using ParallelDo. I have 10000 jobs, each of which returns a 1M output, which I would like saved to a file on the head machine, one file per run. My basic challenge is that Mathematica saves by default to the slave node.
I have many solutions, none of which very good:
1-Save in a temp directory on slave, then collect the files at the end of the job. My best option so far, but not very elegant. I pass the file names as variables, save those to a file, and write a shell script to copy the files to the local machine.
2-Pass output as variables to the head node. The problem is that this fills up the memory of the master machine and saves nothing until the 10000 jobs are running. I can break the job into chunks by hand, but that's not efficient.
Is there a way to either:
-Pass the output as a stream-like object that can be saved on the fly, rather than at the end of the computation,
-Save efficiently from the slave to the master computer.
-easily collect all the output from the local temp directories?