Building a convenient package distribution system

Question

The PacletManager is nice, but it's not really a low-effort system to work with.

Installing a package can be made pretty easy, but only if the location from which it is to be installed makes it so.

For instance, I set up my paclet servers so that I can do things like this:

PacletInstall[pacletName,
 "Site"->pacletServer
 ]

but it's not really feasible to expect everyone to do that. It's not that hard, but it requires enough work that most people just won't.

If I'm working with a group of people and we want to share our packages with eachother seamlessly, making everyone make and maintain a paclet server won't cut it.

I know it shouldn't really be our responsibility to implement an easy-to-use package index/manager but I'm also a little tired of waiting for WRI to make it exist.

I've identified 5 criteria that I think would make for a decent foundation for a package ecosystem:

1. Installing a package is a one-line process
2. Figuring out what packages are available for download is a one-line process
3. The package index can be interacted with as a GUI
4. Making one's own package available for download can either in one line or via a simple form
5. No knowledge of paclets or the paclet manager is necessary to use the system

How could we implement such a system? (should we even implement such a system?)

Alternatively, does anyone else have a better low-(user)effort way to seamlessly share packages as a group?

Note that this is aimed at larger-scale packages. Small snippets can be shared without much trouble.

Answer 1

Preamble

A few notes

Here’s a system I’ve been working on that hits most of my targets.

Note that until I have time to test it on another machine it should be considered only a proto-type solution.

At this stage I expect the system to be very buggy, given the quantity of code that is backing it and the fact that I only have one old Mac to test things on. Feedback will be appreciated and bug fixes will be put in place as soon as I have time.

Idea

I decided to GUI-ify my workflow so that it’s easier for others to use.

My idea was to automate the generation of paclet servers so that others can easily use them. At the moment, I only support all of the web-page server capabilities in the cloud or with GitHub pages, but since I also support local servers, any shared-drive server with local-support (Google Drive, OneDrive, Dropbox).

---

Overview:

The entire interface looks like this:

It supports installing and updating paclets from paclet servers, adding paclets to / removing paclets from a paclet server, making new paclet servers, deploying a paclet server to the web or to GitHub, and creating websites for paclet servers.

---

GUI Breakdown

Primary Header

This allows you to easily open a paclet server and allows you to pick which server you want to work with. The setter-bar at the bottom allows you to choose different panes of functionality.

‌

Main Tab

Paclet Listing

This provides a listing of the paclets available on your server. You can click "Open Interface" to get the result of the function PacletServerInterface . Double clicking on a name will automatically open that page in the interface. For local paclet servers, it also provides a checkbox to switch between seeing what’s on the cloud server and what’s local.

Paclet Interface

This is the installation and information gathering workhorse. It provides the same listing of paclets, but also provides extra info, like the paclet version, a mailto link for the creator email (if provided), and a button for installing / updating the paclet (only enabled if the paclet is not installed locally with the max paclet version number).

Clicking on a paclet name (or double clicking on the name in the listing) will open up the info page for that paclet:

Add / Remove Buttons

These allow one to add a paclet to the server or remove it.

"Add Paclet" opens up an interface that lets one add a file or directory (depending on whether "Add File" is checked). The paclet gets packed automatically, with the appropriate generation of the "PacletInfo.m" . This then gets added to the server (if local) or uploaded to the cloud.

The changes are then reflected in the server listing.

Site Tab

Edit Server Interface

These input fields allow one to change the base location, name, etc. of the server. These mostly just change where it resides / how it gets copied to the web.

The "Cloud Account" field provides the option to set a cloud account for those of us with many accounts. Account passwords are by default stored in an encoded keychain system I wrote up.

Delete Server Interface

Servers can be deleted with the "Delete Server" interface. It provides the option to delete simply the server configuration ( "server entry" ), the server directory (for local servers), and / or the server as it resides on the cloud.

Publish Pane

For local servers, the "Publish Server" button copies the server to the cloud. This is how changes will generally be made accessible to others.

"Publish Server" also supports generating a web-site for the server, similar to the one found here .

That happens when "Make Pages" is set. "Open Site Config" allows one to change how this site is generated, including providing a different theme or name for the site.

For shared drive servers, no publishing will be necessary.

A GitHub repository for the server can be automatically set up using "Configure GitHub Repo" . This can then be published with "Publish GitHub Server" .

If "Make Pages" is set, it configures a website in the /docs folder.

From there, we can set up a site for the repo as by going to the settings tab for the repo and designating that a site should be built.

When that is done, we get something like this .

New Tab

The new server interface provides fields for adding a new server configuration to the list of known servers. This could be useful, for instance, if a friend or coworker sent you a link to their paclet server.

Alternatively, one can make multiple servers, for public and private use, for storing one type of paclets or another, etc.

---

Discussion

Use Cases

Paclet Distribution

The primary reason for building something like this is just to make it really easy to set up a paclet server. For a Dropbox account in a shared drive it becomes really easy to upload one’s changes.

GitHub integration also obviates the need for a cloud account.

Paclet Installation

The other reason for something like this is that it allows paclets to easily be installed and updates. It’s a matter of adding the server website to one’s configurations (via the "New" interface) and then just opening up the "Main" interface and clicking a few buttons.

Updates and de-installation are similarly simple.

Paclet Sites

By allowing one to build out a site for their paclets, it becomes possible to make nice interfaces for code sharing and publishing. This feature isn’t 100% complete, as my markdown parser isn’t 100% complete, but it’s good enough for many things (images, animations, links, etc.).

Issues

Robustness

This builds on top of ~6 major packages I’ve developed, but until others put these through their paces I won’t know where all the bugs are.

I use most of these packages and this kind of workflow every day in my development work, but I’m not sure how robust they really are.

KeyChain Mechanism

I made my own keychain mechanism and authentication dialogs that I use to store passwords and things. My GitHub connection and Cloud stuff take advantage of this. It stores a password encoded collection of passwords and other data at $UserBaseDirectory/ApplicationData/EncodedCache/KeyChain

If you see a dialog that looks like this

You can make use of the keychain mechanism by entering any password (that will then be the password used in encoding and for retrieval) or let me know and I’ll try to track down where it’s using that and figure out a work-around.

No Index

One major design issue is that this doesn’t support a centralized index. I have ideas for how to fix this, such as supporting a read-only paclet server that is defined as a collection of other paclet servers, but until such time as I figure this out, new servers will have to be added using the "New" tab.

---

Installation

Finally, this lives in my standard toolkit, which you can get like so:

PacletInstall[
  "BTools",
  "Site" -> 
  "http://www.wolframcloud.com/objects/b3m2a1.paclets/PacletServer"
  ];
RebuildPacletData[]

Alternatively, I've spun off a smaller sub-paclet from that big one which you can get like so:

PacletInstall[
  "PacKit",
  "Site" -> 
  "http://www.wolframcloud.com/objects/b3m2a1.paclets/PacletServer"
  ];
RebuildPacletData[]

It will then live under Palettes > BTools (or PacKit) > Paclet Server Manager.

Adding my paclet server:

You can add my paclet server by going to the "New" tab and specifying that the "Base Location" is:

http : // www.wolframcloud.com/objects/b3m2a1.paclets

And the "Name" is

PacletServer

Once you do that you’ll see all of my paclets available for installation:

Answer 2

I've now tried to do something much richer than my previous system (and many thanks to Szabolcs and halirutan for the beta testing and advice while getting this off the ground). I talked about the new system here and reposted that on Wolfram Community here

I developed an open, easy install paclet server that anyone can contribute to. The site is hosted on GitHub so it's possible to see everything I've put into it and we can use all of git's tools for collaboration.

If you're interested in the site it's here: https://paclets.github.io/PacletServer/

Here're some nice features of the site:

• Single-line package installation (curtesy of the PacletManager)
• Complete change log so we can trust that whoever claimed to make the package really did
• (semi) rich information pages for each paclet on the site e.g. this one
• Complete installation instructions on a paclet page
• An interface paclet so that you can submit your packages in two lines of code

Building out the site has taken a good amount of time and energy (and there's still lots of work to do), but my hope is that the convenience it provides will convince people to make their code available to the community and in turn the community will download, use, and extend it.

That type of collaborative environment can only improve Mathematica as a system.