4
$\begingroup$

I am attempting to parallelize the construction of a table using MATLink and ParallelTable.

Let's consider a sample MATLAB function defined by

    function [out1,out2] = TestFunction(in1,in2,in3)
    %TestFunction Example function
    out1=in1;
    out2=in2+in3;
    end

Consider the non-parallel construction of the table

    Needs["MATLink`"];
    OpenMATLAB[];
    TestFunction = MFunction["TestFunction", "OutputArguments" -> 2];
    tab = Table[
       {out1, out2} = TestFunction[i, 2 i, 3 i];
       {out1, out2}
       , {i, 1, 3}];

Which yields the output

   {{1.,5.},{2.,10.},{3.,15.}}

My attempt to parallelize the construction is as follows:

First, launch kernels and load MATLink:

   LaunchKernels[];
   ParallelNeeds["MATLink`"];
   ParallelEvaluate[
      OpenMATLAB[];
      TestFunction = MFunction["TestFunction", "OutputArguments" -> 2];
    ];

Which yields the following two errors for each of the kernels

   Global`OpenMATLAB::shdw :  Symbol OpenMATLAB appears in multiple contexts   {Global`,MATLink`}; definitions in context Global` may shadow or be shadowed by other definitions.
   Global`MFunction::shdw :  Symbol MFunction appears in multiple contexts {Global`,MATLink`}; definitions in context Global` may shadow or be shadowed by other definitions.

and MATLAB is not opened on any kernals. As a result, the parallel table construction

   tab = ParallelTable[
      {out1, out2} = TestFunction[i, 2 i, 3 i];
      {out1, out2}
      , {i, 1, 3}];

yields the error

Set::shape :  Lists {out1,out2} and Global`MFunction[TestFunction,OutputArguments->2][1,2,3] are not the same shape.

for each kernel. No output is generated.

What is the correct way to use MATLink on multiple kernels?

Additional details:

The output of FindFile["Matlink"]` is

   In[1]:=FindFile["MATLink`"]
   Out[1]=C:\Users\VHYDAN001\AppData\Roaming\Mathematica\Applications\MATLink\Kernel\init.m

The output of ParallelEvaluate[FindFile["MATLINK"]` is

   In[2]:=ParallelEvaluate[FindFile["MATLink`"]]
   Out[2]={C:\Users\VHYDAN001\AppData\Roaming\Mathematica\Applications\MATLink\Kernel\init.m,
   C:\Users\VHYDAN001\AppData\Roaming\Mathematica\Applications\MATLink\Kernel\init.m,
   C:\Users\VHYDAN001\AppData\Roaming\Mathematica\Applications\MATLink\Kernel\init.m,
   C:\Users\VHYDAN001\AppData\Roaming\Mathematica\Applications\MATLink\Kernel\init.m}

The output of $Path is

   In[3]:= $Path
   Out[3]= {C:\Program Files\Wolfram Research\Mathematica\11.1\AddOns\Applications,
   C:\Program Files\Wolfram Research\Mathematica\11.1\SystemFiles\Links,
   C:\Users\VHYDAN001\AppData\Roaming\Mathematica\Kernel,
   C:\Users\VHYDAN001\AppData\Roaming\Mathematica\Autoload,
   C:\Users\VHYDAN001\AppData\Roaming\Mathematica\Applications,
   C:\ProgramData\Mathematica\Kernel,
   C:\ProgramData\Mathematica\Autoload,
   C:\ProgramData\Mathematica\Applications,
   .,
   C:\Users\VHYDAN001,
   C:\Program Files\Wolfram Research\Mathematica\11.1\AddOns\Packages,
   C:\Program Files\Wolfram Research\Mathematica\11.1\SystemFiles\Autoload,
   C:\Program Files\Wolfram Research\Mathematica\11.1\AddOns\Autoload,
   C:\Program Files\Wolfram Research\Mathematica\11.1\AddOns\Applications,
   C:\Program Files\Wolfram Research\Mathematica\11.1\AddOns\ExtraPackages,
   C:\Program Files\Wolfram Research\Mathematica\11.1\SystemFiles\Kernel\Packages,
   C:\Program Files\Wolfram Research\Mathematica\11.1\Documentation\English\System,
   C:\Program Files\Wolfram Research\Mathematica\11.1\SystemFiles\Data\ICC}

The output of First@ParallelEvaluate[$Path] is

   In[4]:= First@ParallelEvaluate[$Path]
   Out[4]= {C:\Program Files\Wolfram Research\Mathematica\11.1\SystemFiles\Links,
   C:\Users\VHYDAN001\AppData\Roaming\Mathematica\Kernel,
   C:\Users\VHYDAN001\AppData\Roaming\Mathematica\Autoload,
   C:\Users\VHYDAN001\AppData\Roaming\Mathematica\Applications,
   C:\ProgramData\Mathematica\Kernel,
   C:\ProgramData\Mathematica\Autoload,
   C:\ProgramData\Mathematica\Applications,
   .,
   C:\Users\VHYDAN001,
   C:\Program Files\Wolfram Research\Mathematica\11.1\AddOns\Packages,
   C:\Program Files\Wolfram Research\Mathematica\11.1\SystemFiles\Autoload,
   C:\Program Files\Wolfram Research\Mathematica\11.1\AddOns\Autoload,
   C:\Program Files\Wolfram Research\Mathematica\11.1\AddOns\Applications,
   C:\Program Files\Wolfram Research\Mathematica\11.1\AddOns\ExtraPackages,
   C:\Program Files\Wolfram Research\Mathematica\11.1\SystemFiles\Kernel\Packages,
   C:\Program Files\Wolfram Research\Mathematica\11.1\Documentation\English\System,
   C:\Program Files\Wolfram Research\Mathematica\11.1\SystemFiles\Data\ICC}
$\endgroup$
3
  • $\begingroup$ Interesting. Is a new MATLAB instance opened for each parallel Mathematica kernel? Isn't licensing an issue? $\endgroup$ – Gustavo Delfino Feb 21 '20 at 16:15
  • $\begingroup$ @GustavoDelfino I want to open a new MATLAB instance for each parallel kernel but I am struggling to do so. If I'm just using MATLAB (without MATLink or Mathematica) I am able to open multiple instances of MATLAB. Further, if I open more than one instance of Mathematica I am able to open MATLAB from each instance and I can run the non-parallel code calling the MATLAB function from each instance of Mathematica simultaneously. So I don't think licensing is an issue. $\endgroup$ – DvanHuyssteen Feb 22 '20 at 7:59
  • $\begingroup$ @GustavoDelfino No, the Global context makes it clear that the package was never loaded on subkernels. I will be EXTREMELY surprised if the situation is not what I described in my answer. $\endgroup$ – Szabolcs Feb 22 '20 at 16:59
2
$\begingroup$

The package was loaded on the subkernels, but not the main kernel. ParallelNeeds only loads it on the subkernels. You must also use Needs.

What happened is that since the package wasn't loaded on the main kernel, the symbol OpenMATLAB was interpreted by the main kernel as Global`OpenMATLAB, not has MATLink`OpenMATLAB. It sent the Global` version to the subkernels, which then issued a warning about multiple symbols with the same name in different contexts.

Please try like this:

LaunchKernels[];
Needs["MATLink`"];
ParallelNeeds["MATLink`"];   
ParallelEvaluate[
   OpenMATLAB[];
   TestFunction = MFunction["TestFunction", "OutputArguments" -> 2];
];
$\endgroup$
6
  • $\begingroup$ I'm sorry, I'm not well today and can't show you how to effectively set the path on parallel kernels. For now just use ParallelEvaluate to change $Path. $\endgroup$ – Szabolcs Feb 21 '20 at 13:54
  • $\begingroup$ Executing ParallelEvaluate[Print@FindFile["MATLink"]] yields a path that is not on the $Path for the parallel kernels. However, it also isn't on the regular $Path and for non parallel computations I am able to use MATLink. Adding the path to each kernel does eliminate the first error messages, but I am still unable to open MATLAB using ParallelNeeds["MATLink"]; ParallelEvaluate[ OpenMATLAB[]; TestFunction = MFunction["TestFunction", "OutputArguments" -> 2];]; $\endgroup$ – DvanHuyssteen Feb 22 '20 at 8:02
  • $\begingroup$ @DvanHuyssteen There are some misunderstandings here. What you say cannot possibly be true, at least the way I understand it. So, please edit into your answer the output from: FindFile["MATLink`"], ParallelEvaluate[FindFile["MATLink`"]], $Path, First@ParallelEvaluate[$Path]. Evaluate these (and nothing else!) as the first four commands immediately after kernel startup. $\endgroup$ – Szabolcs Feb 22 '20 at 16:58
  • $\begingroup$ I have added these outputs to the original question under the Additional Details section. $\endgroup$ – DvanHuyssteen Feb 23 '20 at 7:33
  • 1
    $\begingroup$ @DvanHuyssteen Thanks for posting the results. You say "Which yields the following two errors for each of the kernels". Actually this looks like you did not load MATLink on the main kernel before trying to use it on subkernels. You have the ParallelNeeds, but you are missing the Needs. Make sure you use both Needs and ParallelNeeds. Let me know if this was the problem. $\endgroup$ – Szabolcs Feb 23 '20 at 9:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.