0
$\begingroup$

Suppose I have a reasonably large number of floating point arrays, obtained from prior numerical simulations. These are stored as binary files named data_#####.dat where ##### is the number used as input parameter to my simulation. The files need to be post-processed to obtain some physical data, say area and mass. I currently have a notebook to each file which, schematically, does the following

processFile[x_] := Block[{ ...some local variables... },
    importedData = Import[
        "data_"<>ToString[x]<>".dat",
        "Real64", ByteOrdering -> 1, "HeaderBytes" -> 8];

    data[x]["area"] = calculateArea[importedData];
    data[x]["mass"] = calculateMass[importedData];

    x
];

xList = processFile /@ Table[i, {i, 1, 100}];

This works pretty well: after the evaluation terminates I can just query things like data[30]["mass"] to get the mass of the n = 30 solution, or data[87]["area"] to query the area of the n = 87 solution. I can even DumpSave["processed.mx", data] for later usage.

Note that, in this process, Definition[data] contains definitions only for data[x]["mass"] and data[x]["area"], and not data[x].

The issue is that the actual notebook calculates not just two, but rather around ten physical quantities, per file, and each calculation takes a nontrivial amount of time. It is therefore desirable to parallelize this process.

The obvious thing to do is the change the last line to

xList = ParallelMap[processFile, Table[i, {i, 1, 100}]];

but, since the results are "stored" as definitions in the data symbol, we must first declare it as a shared function

SetSharedFunction[data]

However, this does not work.

While SetSharedFunction would allow me to do assignments like data[x] = someValue in each parallel subkernel (indeed this is what the example does in the documentation), data[x]["mass"] = someValue fails because the subkernel first evaluates data[x] to Null, and obviously Mathematica protests at my attempt to assign a value to Null["mass"].

Is there a way for me to do what I want to do here?

$\endgroup$
  • 1
    $\begingroup$ Why don't you just use a data structure that can be passed around freely (Association?) instead of attaching everything to a hard-coded name like data? $\endgroup$ – Szabolcs Nov 17 '14 at 18:47
  • $\begingroup$ This was exactly what I was looking for before going with the "attching to data" route. Honestly, in my years of using mma I've somehow never come across this rather fundamental built-in. I did try searching for "HashMap in Mathematica" or some such earlier but somehow that yielded nothing useful. Thanks a lot! $\endgroup$ – Saran Nov 17 '14 at 18:49
  • $\begingroup$ Also, Association is clearly an appropriate solution here which I will indeed use. However, the question still holds: is there a way to define to a symbol in the way described above from parallel kernels? $\endgroup$ – Saran Nov 17 '14 at 18:59
  • $\begingroup$ Association is new in version 10. You can use a workaround similar to SetSharedFunction[f]; f[prop_, x_] := data[prop][x] = x, but do not use SetSharedFunction on data in this case. SubValues don't seem to be supported by SetSharedFunction. $\endgroup$ – Szabolcs Nov 17 '14 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.