LetL Breaks in ParallelEvaluate

Question

This was a wild head-scratcher for me to hunt down, but I now am sure LetL is the culprit. Here's a small example:

ParallelEvaluate[LetL[{id = $KernelID, sq = id*id}, sq]]

which gives

With: Variable 1 in local variable specification {1} requires a value.

With: Variable 2 in local variable specification {2} requires a value.

With: Variable 3 in local variable specification {3} requires a value.

General: Further output of With::lvws will be suppressed during this calculation

Out[1] = {With[{1}, With[{1}, 1]], With[{2}, With[{4}, 4]], With[{3}, With[{9}, 9]], With[{4}, With[{16}, 16]]}

A similar failure happens with

ParallelTable[LetL[{x = i j}, x^2], {i, 1, 3}, {j, 1, 3}]
ParallelMap[LetL[{f = #^2}, Sqrt[f]] &, Range[1, 10]]

Question: Can we formulate LetL in a way that it performs correctly inside a ParallelEvaluate, ParallelTable, and other Parallel functions?

Partial answer: Put LetL in its own package and evaluate ParallelNeeds["LetL`"]. Then the above examples work correctly. I use LetL enough that I have it in my auto-loaded packages, and I don't always want to launch multiple kernels on startup.

Alternative question to solve my use case: How can I get LetL to be ParallelNeedsed when additional kernels are launched?

Answer 1

The problem

Apparently, the problem is not with Let itself, but with the fact that the symbol's definitions are not autoloaded, as you noted youself . So what happens then is that the HoldAll attribute of Let is being ignored by the sub-kernels, and when you get the result back for evaluation on the master kernel, it is too late.

Possible solution

What I will suggest below may be a bit heavy-handed, but it should work. You can convert the code into a paclet. The repository with working code is here, where you can look for details of how to set things up so the symbol gets autoloaded.

You can download and extract the contents into applications folder, but I have also added a bootstrap installer which would do it for you, for convenience. The following code needs to be executed just once (note the call to Quit at the end):

Import[
   "https://raw.githubusercontent.com/lshifr/LetWL/master/BootstrapInstall.m"
]

RebuildPacletData[]

Quit

After which it works just fine:

ParallelEvaluate[LetWL`Let[{id = $KernelID, sq = id*id}, sq]]

(* {1, 4, 9, 16} *)

Note that I have changed the context of Let to be LetWL`. Otherwise, the code of implementation of Let is pretty much the same as your original one.