Clean package update for API/FormFunctions on Wolfram Cloud

Question

Cross posted on community.wolfram.com (new comments)

TLDR

There is a pool of kernels/sessions available for each user, you don't have control over the pool only over specific kernel you currently evaluate in. Re-upload of a package should be followed by Quit[] in every kernel where previous version of a package was used. I failed to find tools for that.

Moreover, you can Quit[] via browser notebook interface but can't via CloudEvaluate or APIFunction so there is no way to clear specific kernel used by APIFunction.

Background

I am developing a package which I want to deploy to my Wolfram Public Cloud's account and I want a set of APIFunctions/FormFunctions/etc to be able to use it.

• PacletManager has limited functionality, so I am just uploading the package archive and extract it to WPC's $UserBaseDirectory / Applications /.

• APIFunctions and friends have the form:

   APIFunction[{}, (Needs["TestPackage`"];Symbol["TestPackage`api2"][...]) &]
  

  I use Symbol because otherwise TestPackage` definitions are uploaded and I want to avoid that. There should be one code source, the package in $UserBaseDirectory.

Problem

The problem is that kernel management on WPC is a closed black box. If you call the API twice, each time it uses one of the kernels from the pool. This also applies to CloudEvaluate etc.

$UserBaseDirectory is shared, but Get will find the current version only for the current kernel.

If in another kernel TestPackage` was loaded earlier, Needs will not load the current version there :-/ I do not want to use Get in my APIFunctions, the proper way is to call Needs.

And there is no way to Quit each available kernel.

Example

is worth 10^3 words:

First we will mimic a package upload, 10 times. So it was uploaded, something was fixed, it was uploaded again, etc:

packageTemplate = StringTemplate["
   BeginPackage[\"TestPackage<*\"`\"*>\"];
   myValue = \"``\"
   EndPackage[];
"];

Do[
   CloudExport[packageTemplate@RandomReal[], "Text", "TestPackage.m"]
 ; CloudEvaluate[
       CopyFile[
           "TestPackage.m"
         , StringRiffle[
               {$UserBaseDirectory, "Applications", "TestPackage.m"}
             , "/"
           ]
         , OverwriteTarget -> True
       ]
     ; Get @ "TestPackage`"
   ]
 , {10}
]

Now let's call the code based on the package 25 times:

Table[
    CloudEvaluate[
        Needs["TestPackage`"]
      ; {$SessionID, Symbol["TestPackage`myValue"]}
    ]
 ,  {25}
] // CountBy[Last] // Normal // Column

7 different results, I expect 1, the value from the 10th deployment!

The question

How to upload a package properly, as we can see Get after the upload only affects one particular kernel. CloudEvaluate@Quit[] will not help either.

How to reset them all?

Requirements

 APIFunction[{}, (Get["TestPackage`"];Symbol["TestPackage`api2"][...]) &]

could solve it but if the package contains Protected/Locked symbols you will get a flood of errors. And because of the same reason you can't ClearAll symbols. And obviously you can't Quit in APIFunction.

Further reading

The problem was known but not diagnosed already 3 years ago:

http://community.wolfram.com/groups/-/m/t/295756

Answer 1

Edit:

For more on this Context trick see this: Importing an mx file into a context

And just as full disclosure here's a note on it circumventing Protected: Mr. Wizard discovers context reassignment

Original:

So we can circumvent this with the Context[x]="new`" trick, I think, plus a versioning file to track things.

We'll set up a $uploadVersion symbol and an uploadPackage package function, but otherwise we won't change things much to start:

If[! IntegerQ@$uploadVersion, $uploadVersion = 1];

packageTemplate =
  StringTemplate["
      BeginPackage[\"TestPackage<*\"`\"*>\"];
      $uploadVersion= ``;
      myValue = \"``\";
      EndPackage[];
      Protect[\"TestPackage<*\"`\"*>\"];
      SetAttributes[Evaluate@Names[\"TestPackage<*\"`\"*>\"],{Locked}];
   "];

uploadPackage[] :=
  (
   $uploadVersion++;
   CloudExport[
    packageTemplate[$uploadVersion, RandomReal[]],
    "Text",
    "TestPackage.m"
    ];
   CloudExport[
    $uploadVersion,
    "Text",
    "$TestPackageVersion.m"
    ];
   CloudEvaluate[
    CopyFile["TestPackage.m", 
     FileNameJoin[{$UserBaseDirectory, "Applications", 
       "TestPackage.m"}], OverwriteTarget -> True
     ];
    CopyFile["$TestPackageVersion.m", 
     FileNameJoin[{$UserBaseDirectory, "Applications", 
       "$TestPackageVersion.m"}], OverwriteTarget -> True
     ];
    ]
   );

Then we'll upload and screw things up:

Do[uploadPackage[]; CloudEvaluate[Quiet@Get@"TestPackage`"], 10];

Table[
  CloudEvaluate[
   Needs["TestPackage`"];
   With[{old = 
      Symbol["TestPackage`$uploadVersion"] =!= 
       Get@"$TestPackageVersion`"},
    {
     old,
     Symbol["TestPackage`myValue"],
     Symbol["TestPackage`$uploadVersion"]
     }
    ]
   ],
  {25}
  ] // TableForm

True    0.927189    110
True    0.927189    110
True    0.0370685   113
True    0.593577    117
True    0.927189    110
True    0.927189    110
True    0.62374 114
True    0.927189    110
True    0.444576    109
True    0.402844    115
True    0.14487 119
True    0.927189    110
True    0.927189    110
False   0.590517    120
True    0.927189    110
True    0.927189    110
True    0.927189    110
True    0.927189    110
True    0.0370685   113
True    0.593577    117
True    0.927189    110
True    0.927189    110
True    0.62374 114
True    0.317602    118
True    0.927189    110

The last thing there is the upload version, the first thing is whether the package should be reloaded, and the second is the value.

So now we'll use the Context trick I discussed and resynchronize:

Table[
  CloudEvaluate[
   Needs["TestPackage`"];
   With[{old = 
      Symbol["TestPackage`$uploadVersion"] =!= 
       Get@"$TestPackageVersion`"},
    If[Symbol["TestPackage`$uploadVersion"] =!= 
      Get@"$TestPackageVersion`",
     Map[
      Quiet[
        Context[#] = ("TestPackage`" <> "Version" <> 
           ToString@Symbol["TestPackage`$uploadVersion"] <> "`")
        ] &,
      Names["TestPackage`*"]
      ];
     Get["TestPackage`"]
     ];
    {
     old,
     Symbol["TestPackage`myValue"],
     Symbol["TestPackage`$uploadVersion"]
     }
    ]
   ],
  {25}
  ] // TableForm

True    0.590517    120
True    0.590517    120
True    0.590517    120
True    0.590517    120
True    0.590517    120
True    0.590517    120
True    0.590517    120
True    0.590517    120
True    0.590517    120
True    0.590517    120
True    0.590517    120
True    0.590517    120
True    0.590517    120
True    0.590517    120
True    0.590517    120
True    0.590517    120
False   0.590517    120
False   0.590517    120
False   0.590517    120
False   0.590517    120
False   0.590517    120
True    0.590517    120
False   0.590517    120
False   0.590517    120
False   0.590517    120

Note that the Context trick really is crucial here as if we reload and test without it:

Do[
  uploadPackage[];
  CloudEvaluate[
   Map[
    Quiet[
      Context[#] = ("TestPackage`" <> "Version" <> 
         ToString@Symbol["TestPackage`$uploadVersion"] <> "`")
      ] &,
    Names["TestPackage`*"]
    ];
   Quiet[Get@"TestPackage`"]
   ], 10];

    Table[
      CloudEvaluate[
       Needs["TestPackage`"];
       With[{old = 
          Symbol["TestPackage`$uploadVersion"] =!= 
           Get@"$TestPackageVersion`"},
        If[Symbol["TestPackage`$uploadVersion"] =!= 
          Get@"$TestPackageVersion`",
         Get["TestPackage`"]
         ];
        {
         old,
         Symbol["TestPackage`myValue"],
         Symbol["TestPackage`$uploadVersion"]
         }
        ]
       ],
      {25}
      ] // TableForm


True    0.731728    140
True    0.731728    140
True    0.0907238   167
True    0.731728    140
True    0.731728    140
True    0.731728    140
True    0.815141    168
True    0.731728    140
True    0.2475  165
True    0.104199    169
True    0.731728    140
True    0.731728    140
True    0.458742    166
False   0.235427    170
True    0.731728    140
True    0.853991    162
True    0.731728    140
True    0.405875    163
True    0.731728    140
False   0.235427    170
True    0.0907238   167
True    0.731728    140
True    0.731728    140
True    0.731728    140
True    0.815141    168

We get out a bunch of messages (due to Protected and Locked) and the versions aren't aligned.