3
$\begingroup$

I have created an entity store for financial data, as follows:enter image description here

It generally works well, except that when I try to use parallelization, it fails. For example:

first distribute the definition of the store:

DistributeDefinitions[StockDataStore]

Then try to use ParallelTable:

    perfData =  ParallelTable[
   Performance[stock, formationDate, 
    currentDate, {60, 12, 9, 6, 3}], {stock, universe[[1 ;; 20]]}];


            Missing["UnknownType", 
             "Stocks"], 

etc etc

The code works fine if I use the regular Table function.

It seems as if parallelization doesn't work with EntityStores. or am I missing something?

$\endgroup$
2
  • 2
    $\begingroup$ You would need to register the entity store on all parallel kernels. $\endgroup$
    – Jason B.
    Oct 29, 2022 at 17:52
  • $\begingroup$ Yes that works. Thank you. $\endgroup$
    – MMAUser
    Oct 30, 2022 at 9:03

2 Answers 2

2
$\begingroup$

Converting comment into an answer for posterity:

It is not enough to distribute the definition of the new EntityStore to all parallel kernels, you must register the store so that EntityValue can know about it.

DistributeDefinitions[StockDataStore]
ParallelEvaluate[EntityRegister[StockDataStore]]
$\endgroup$
1
  • $\begingroup$ I am going to accept this answer, because it identified the problem and set me on the right path. But it's achingly slow, because DistributDefinitions takes an age to distribution the entity store to all the parallel kernels. In my case I gave up waiting more than 20 mins! There is a much faster procedure that I have given in my own answer. $\endgroup$ Nov 11, 2022 at 12:08
0
$\begingroup$

Jason B. identified the key issue, which is that the entity store needs to be defined on each parallel kernel before it can be registered. But using DistributeDefinitions is achingly slow.

A much faster procedure is as follows:

  1. Download the Entity Store in each kernel (in this case from an mx file, for speed):

    ParallelEvaluate[ Get[StringJoin[dirEntityStore, "EquitiesEntityStore.mx"]]]

  2. Then register the store on each kernel, as before:

    ParallelEvaluate[EntityRegister[EquitiesEntityStore]]

    {{"Equities"}, {"Equities"}, {"Equities"}, {"Equities"},
    {"Equities"}, {"Equities"}, {"Equities"}, {"Equities"}, {"Equities"},
    {"Equities"}, {"Equities"}, {"Equities"}, {"Equities"}, {"Equities"},
    {"Equities"}, {"Equities"}, {"Equities"}, {"Equities"}, {"Equities"},
    {"Equities"}}

NOTE: the important thing is to download and register the Entity Store on the parallel kernels BEFORE you do it on the main kernel.

I suspect this may be because if you register the store on the main kernel first, MMA will resort to using DistributeDefintions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.