Installing a Paclet from GitHub (or other non-paclet server site)

Question

How can I install a package from GitHub as a paclet?

Can such a process be formulated in general?

Answer 1

Preamble

Note that this is a special case. If the developer provides you with a paclet file already you can just install that by URL. See this for more

This is all built into my PackageData.net service connection. You can get that here or by running

PacletInstall["ServiceConnection_PackageData",
 "Site"->"http://www.wolframcloud.com/objects/b3m2a1.paclets/PacletServer"
 ]

If you just want to look at the package itself it's on GitHub https://github.com/b3m2a1/mathematica-tools

The basic strategy is this:

• Download the files (for GitHub, check for releases / clone the repo)
• Check the directory layout to see if it's already a paclet. If not, rebuild the directory slightly so that it can be treated as such.
• Call PackPaclet then PacletInstall

---

Case 1: Properly Formatted Paclets

So if the repo is laid out properly as a paclet this is easy.

Simply clone the repo:

With[{
  dir =
   FileNameJoin@{
     $TemporaryDirectory,
     DeleteCases[URLParse[loc, "Path"], ""][[2]]
     }
  },
 Quiet[
  DeleteDirectory[dir, DeleteContents -> True];
  CreateDirectory[dir]
  ];
 RunProcess[{"git", "clone", loc, dir}];
 dir
 ]

Call PackPaclet on it

And call PacletInstall on that

Case 1.1: Releases

But maybe the creator of the repository provides releases for us (e.g. Szabolcs). If that's the case, we'll want to find the most recent of these for release and use that. So we can generalize this pull process:

gitPacletPull[loc : (_String | _URL | _File)] :=

  If[StringContainsQ[URLParse[loc, "PathString"], "releases"],
   Block[{
     release =
      SelectFirst[
       Flatten@{
         Import[
          URLBuild@
             ReplacePart[#, {
               "Domain" -> "api.github.com",
               "Path" ->
                With[{cleanpath = DeleteCases[#["Path"], ""]},
                 If[cleanpath[[1]] === "repos",
                  cleanpath,
                  Prepend[cleanpath, "repos"]
                  ]
                 ]
               }] &@URLParse[loc],
          "RawJSON"]},
       KeyMemberQ[#, "assets"] || 
         KeyMemberQ[#, 
           "zipball_url"] &&
            (#["prerelease"] =!= True) &
       ]
     },
    If[AssociationQ@release,
     With[{tmp = CreateDirectory[]},
      First@MinimalBy[FileNameDepth]@Select[DirectoryQ]@
         If[
          KeyMemberQ[release, "assets"] && 
           Length@release["assets"] > 0,
          With[{url =
             release[["assets", -1, "browser_download_url"]]
            },
           ExtractArchive[
            URLDownload[url,
             FileNameJoin@{
               $TemporaryDirectory,
               URLParse[url, "Path"][[-1]]
               }
             ],
            tmp
            ]
           ],
          ExtractArchive[
           URLDownload[
            release["zipball_url"],
            FileNameJoin@{
              $TemporaryDirectory,
              URLParse[release["zipball_url"], "Path"][[-1]]
              }
            ],
           tmp
           ]
          ]
      ],
     $Failed
     ]
    ],
   With[{
     dir =
      FileNameJoin@{
        $TemporaryDirectory,
        DeleteCases[URLParse[loc, "Path"], ""][[2]]
        }
     },
    Quiet[
     DeleteDirectory[dir, DeleteContents -> True];
     CreateDirectory[dir]
     ];
    RunProcess[{"git", "clone", loc, dir}];
    dir
    ]
   ];

---

Case 2: Unformatted Paclet

If not, we'll need to auto-detect the layout. There's a lot of code here (much of it treating various special cases), but it's all worth having. Basically it tries to detect the paclet layout and, if it fails to do so, makes the layout appropriate for a paclet. The corner case of just having all the code in a notebook is just to make a package that'll reopen a copy of that notebook when the paclet is called.

Note that it calls things like PDexpressionBundle which auto-create a "PacletInfo.m". They're essentially just copies of the definitions here.

Options[installPacletGenerate] = {
   "Verbose" -> False
   };

installPacletGenerate[dir : (_String | _File)?DirectoryQ, 
   ops : OptionsPattern[]] :=
  Block[{bundleDir = dir},
   If[OptionValue@"Verbose",
    DisplayTemporary@
     Internal`LoadingPanel[
      TemplateApply["Bundling PDpaclet for ``", dir]
      ]
    ];
   (* ------------ Extract Archive Files --------------- *)

   If[FileExistsQ@#, Quiet@ExtractArchive[#, dir]] & /@
    Map[
     FileNameJoin@{dir, FileBaseName@dir <> #} &,
     {".zip", ".gz"}
     ];
   (* ------------ Detect Paclet Layout --------------- *)
   Which[
    FileExistsQ@FileNameJoin@{dir, "PacletInfo.m"},
    bundleDir = dir,
    FileExistsQ@FileNameJoin@{dir, FileBaseName[dir] <> ".m"} ||

     FileExistsQ@FileNameJoin@{dir, FileBaseName[dir] <> ".wl"} ||

       FileExistsQ@FileNameJoin@{dir, "Kernel", "init" <> ".m"} ||

       FileExistsQ@FileNameJoin@{dir, "Kernel", "init" <> ".wl"},
    bundleDir = dir;
    PDpacletExpressionBundle[bundleDir],
    FileExistsQ@FileNameJoin@{dir, FileBaseName@dir, "PacletInfo.m"},
    bundleDir = FileNameJoin@{dir, FileBaseName@dir},
    FileExistsQ@
     FileNameJoin@{dir, FileBaseName@dir, FileBaseName@dir <> ".m"},
    bundleDir = FileNameJoin@{dir, FileBaseName@dir};
    PDpacletExpressionBundle[bundleDir],
    FileExistsQ@FileNameJoin@{dir, FileBaseName[dir] <> ".nb"},
    Export[
     FileNameJoin@{dir, FileBaseName[dir] <> ".m"},
     "(*Open package notebook*)
     CreateDocument[
        Import@
            StringReplace[$InputFileName,\".m\"->\".nb\"]
     	]",
     "Text"
     ];
    bundleDir = dir;
    PDpacletExpressionBundle[bundleDir],
    _,
    Message[PDInstallPaclet::laywha];
    Throw[$Failed]
    ];
   PDpacletBundle[bundleDir]
   ];

installPacletGenerate[file : (_String | _File)?FileExistsQ, 
   ops : OptionsPattern[]] :=
  Switch[FileExtension[file],
   "m" | "wl",
   If[OptionValue@"Verbose",
    DisplayTemporary@
     Internal`LoadingPanel[
      TemplateApply["Bundling PDpaclet for ``", file]
      ]
    ];
   With[{dir =
      FileNameJoin@{
        $TemporaryDirectory,
        FileBaseName@file
        }
     },
    Quiet@CreateDirectory[dir];
    If[FileExistsQ@
      FileNameJoin@{
        DirectoryName@file,
        "DependencyInfo.m"
        },
     CopyFile[
      FileNameJoin@{
        DirectoryName@file,
        "DependencyInfo.m"
        },
      FileNameJoin@{
        dir,
        "DependencyInfo.m"
        }	
      ]
     ];
    If[FileExistsQ@
      FileNameJoin@{
        DirectoryName@file,
        "PacletInfo.m"
        },
     CopyFile[
      FileNameJoin@{
        DirectoryName@file,
        "PacletInfo.m"
        },
      FileNameJoin@{
        dir,
        "PacletInfo.m"
        }	
      ]
     ];
    CopyFile[file,
     FileNameJoin@{
       dir,
       FileNameTake@file
       },
     OverwriteTarget -> True
     ];
    PDpacletExpressionBundle[dir,
     "Name" ->
      StringReplace[FileBaseName[dir],
       Except[WordCharacter | "$"] -> ""]
     ];
    PDpacletBundle[dir,
     "BuildRoot" -> $TemporaryDirectory
     ]
    ],
   "nb",
   With[{dir =
      FileNameJoin@{
        $TemporaryDirectory,
        StringJoin@RandomSample[Alphabet[], 10],
        FileBaseName@file
        }
     },
    Quiet[
     DeleteDirectory[dir, DeleteContents -> True];
     CreateDirectory[dir, CreateIntermediateDirectories -> True]
     ];
    CopyFile[file, FileNameJoin@{dir, FileNameTake@file}];
    installPacletGenerate[dir]
    ],
   "PDpaclet",
   file,
   _,
   Message[PDInstallPaclet::howdo,
    FileExtension@file
    ]
   ];

---

General Case:

And so generally what we'll want is to pull down the paclet with gitPacletPull or another appropriate pull function (I also wrote one for the Wolfram Library Archive) and then format it with installPacletGenerate before calling PacletInstall.

This process is built into PDInstallPaclet

For example:

PackageDataFunction`PDInstallPaclet[
  "https://github.com/b3m2a1/TestPaclet"];
<< TestPaclet`

gives me:

Or

PackageDataFunction`PDInstallPaclet["https://github.com/szhorvat/\
IGraphM/releases"]

{
PacletManager`Paclet[
 "Name" -> "IGraphM", "Version" -> "0.3.91", 
  "MathematicaVersion" -> "10.0+", 
  "Description" -> "IGraph/M \[Dash] the igraph interface for \
Mathematica.", "Creator" -> "Szabolcs Horvát <[email protected]>", 
  "URL" -> "http://szhorvat.net/mathematica/IGraphM", 
  "Thumbnail" -> "Logo.png", 
  "SystemID" -> {
   "MacOSX-x86-64", "Windows-x86-64", "Linux-x86-64", "Linux-ARM"}, 
  "Extensions" -> {{
    "Kernel", "Root" -> ".", "Context" -> "IGraphM`"}, {
    "LibraryLink"}, {
    "Documentation", "MainPage" -> "IGDocumentation"}}, 
  "Location" -> "~/Library/Mathematica/Paclets/Repository/\
IGraphM-0.3.91"]}