Executing local functions remotely

Question

The following

Attributes[remoteExecute] = {HoldAllComplete};
remoteExecute[expr_] := 
 Module[{compressed, 
   ubuntuBox = RemoteConnect[ip, "username", "password"]}, 
  compressed = Compress[Unevaluated[expr]]; 
  ToExpression[
   RemoteRunProcess[ubuntuBox, 
    TemplateApply[
     "wolframscript -format InputForm -code 'Uncompress[\"``\"]'", 
     compressed], "StandardOutput"]]]

is able to execute functions remotely, so long as your function isn't locally defined. So for example if ~/only_on_ubuntu_box is a file that exists only on your remote machine, this function will fail for:

How can one get around this limitation?

Answer 1

If I understand correctly, you have some functions defined locally that are not defined on the remote machine. When you remote evaluate your expression, which includes these functions, the remote evaluation doesn't work properly. So, you want your remote evaluation function to transfer all of the necessary local functions to the remote machine, and then evaluate your expression.

The good news is that this is exactly what CloudEvaluate does, although the remote machine in this case is the WolframCloud and not your remote machine. However, the machinery created to enable CloudEvaluate to work is already present in Wolfram Language. The function that does the heavy lifting for CloudEvaluate is Language`ExtendedFullDefinition. A very simple version using this function would be:

SetAttributes[compressWithDefinitions, HoldFirst];
compressWithDefinitions[expr_] := With[
    {def = Language`ExtendedFullDefinition[expr]},

    Compress @ Unevaluated[
        Language`ExtendedFullDefinition[] = def;
        expr
    ]
]

The compressed string created by this function first loads all necessary definitions needed to evaluate expr, and then evaluates expr.

I will simulate your remote usage by using LocalSubmit, which sends an evaluation to a separate kernel. Here is a function only defined in the local kernel:

g[x_] := x^2

Here are two compress strings that return the downvalues of g when uncompressed:

str1 = Compress @ Unevaluated[DownValues[g]];
str2 = compressWithDefinitions @ DownValues[g];

Now, if we use LocalSubmit with str1 we find that, as expected, the remote kernel has no downvalues for g:

Clear[dv];
With[{s = str1},
    LocalSubmit[
        Uncompress[s],
        HandlerFunctions-><|"TaskFinished"->((dv=#EvaluationResult)&)|>,
        HandlerFunctionsKeys->"EvaluationResult"
    ]
];
Pause[1];
dv

{}

On the other hand, when using str2, the local kernel does have the transferred downvalues for g:

Clear[dv];
With[{s = str2},
    LocalSubmit[
        Uncompress[s],
        HandlerFunctions-><|"TaskFinished"->((dv=#EvaluationResult)&)|>,
        HandlerFunctionsKeys->"EvaluationResult"
    ]
];
Pause[1];
dv

{HoldPattern[g[x_]] :> x^2}

So, I think you should be able to use compressWithDefinitions to achieve your goals.

Note that there are some additional subtleties that can arise if you try to use package functions, where you need to make sure that the compressed string includes contexts when necessary.