0
$\begingroup$

Suppose I have a reasonably large number of floating point arrays, obtained from prior numerical simulations. These are stored as binary files named data_#####.dat where ##### is the number used as input parameter to my simulation. The files need to be post-processed to obtain some physical data, say area and mass. I currently have a notebook to each file which, schematically, does the following

processFile[x_] := Block[{ ...some local variables... },
    importedData = Import[
        "data_"<>ToString[x]<>".dat",
        "Real64", ByteOrdering -> 1, "HeaderBytes" -> 8];

    data[x]["area"] = calculateArea[importedData];
    data[x]["mass"] = calculateMass[importedData];

    x
];

xList = processFile /@ Table[i, {i, 1, 100}];

This works pretty well: after the evaluation terminates I can just query things like data[30]["mass"] to get the mass of the n = 30 solution, or data[87]["area"] to query the area of the n = 87 solution. I can even DumpSave["processed.mx", data] for later usage.

Note that, in this process, Definition[data] contains definitions only for data[x]["mass"] and data[x]["area"], and not data[x].

The issue is that the actual notebook calculates not just two, but rather around ten physical quantities, per file, and each calculation takes a nontrivial amount of time. It is therefore desirable to parallelize this process.

The obvious thing to do is the change the last line to

xList = ParallelMap[processFile, Table[i, {i, 1, 100}]];

but, since the results are "stored" as definitions in the data symbol, we must first declare it as a shared function

SetSharedFunction[data]

However, this does not work.

While SetSharedFunction would allow me to do assignments like data[x] = someValue in each parallel subkernel (indeed this is what the example does in the documentation), data[x]["mass"] = someValue fails because the subkernel first evaluates data[x] to Null, and obviously Mathematica protests at my attempt to assign a value to Null["mass"].

Is there a way for me to do what I want to do here?

$\endgroup$
4
  • 1
    $\begingroup$ Why don't you just use a data structure that can be passed around freely (Association?) instead of attaching everything to a hard-coded name like data? $\endgroup$
    – Szabolcs
    Nov 17, 2014 at 18:47
  • $\begingroup$ This was exactly what I was looking for before going with the "attching to data" route. Honestly, in my years of using mma I've somehow never come across this rather fundamental built-in. I did try searching for "HashMap in Mathematica" or some such earlier but somehow that yielded nothing useful. Thanks a lot! $\endgroup$ Nov 17, 2014 at 18:49
  • $\begingroup$ Also, Association is clearly an appropriate solution here which I will indeed use. However, the question still holds: is there a way to define to a symbol in the way described above from parallel kernels? $\endgroup$ Nov 17, 2014 at 18:59
  • $\begingroup$ Association is new in version 10. You can use a workaround similar to SetSharedFunction[f]; f[prop_, x_] := data[prop][x] = x, but do not use SetSharedFunction on data in this case. SubValues don't seem to be supported by SetSharedFunction. $\endgroup$
    – Szabolcs
    Nov 17, 2014 at 19:14

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.