Programmatically generate packages from notebook files?

Question

Currently, there are two ways I know of to generate a package (.m) file from a notebook file; one is to set the AutoGeneratedPackage option in the Option Inspector, and the other is to use the menu optoin to "Save as" a package. The first doesn't really suit my needs because it creates the auto-generated packages in the same directory as the notebook file, which clutters things up (there are dozens of notebooks in this directory) and adds a needless complication to keeping the set-up version controlled. The second option is less than ideal because I don't know how to invoke the option programmatically.

In theory, I know it might be possible with FrontEnd commands, but I'd really prefer to be able to do this without a front end if at all possible. This is something I want to do from GNU make, and while I know you can now set up front ends from the command line, ideally I could avoid that.

Answer 1

Not entirely on topic (and probably well-known), but starting with version 6 (I think) you can very comfortably work with the .m files within the frontend. The frontend parses the package and displays it just like any old notebook. You can use sections and do all the fancy typesetting, generate (temporary) output etc. This is also available under File -> New -> Package (.m).

The code is put into "Code" style cells which are Initialization cells, and these are saved (together with text and section information, see Pillsy´s post), stripping any output cells and leaving you with the plain package that works just fine. With the parsing, you also get some helpful formatting and additional gizmos (see image).

So you only maintain one set of packages directly, but with all the benefits of the frontend interface (notice: the parsing can take a few seconds when opening).

And if you are annoyed by the linewrapping behaviour of the package stylesheet, have a look at this thread: Viewing packages in Mathematica

Answer 2

I tried the ExportPacket approach described in Chris Degnen's answer, and while it's very simple, the need for FrontEndExecute and the difficulties dealing with cells of text and section headings and the like make it less appealing. Here's a sketch of the approach I'll ultimately be using, though I haven't filled all the details in yet:

notebookToPackage[notebook_String, package_String] :=
 With[{nbObj = Get@notebook},
  If[
   nbObj === $Failed,
   $Failed,
   Export[package, notebookToStrings@nbObj, {"Text", "Lines"}]]]

This just loads the notebook file and does some basic error handling and exports the result after notebookToStrings converts the mess into strings. Using Get to get the notebook is the easiest way to get it as an object with head Notebook, and from there you can get at the Cell structures.

notebookToStrings[nbObj_] :=
 With[{cells = 
    Cases[nbObj, 
     Cell[body_, style : "Code" | "Text", ___] :> {body, style}, 
     Infinity]},
  Riffle[
   Replace[cells,
    $cellRules, {1}],
   ""]]    

Since notebook objects are Mathematica structures like everything else, you can use the same old structural manipulations and rule replacements that you use everywhere else. Cases proves once again that, with the Infinity level specification, it's the most useful function in the Mathematica library, and the pattern matching means you can whitelist some cells while not having to decide what you want to do about input or output cells or subsections or whatever. Everything else is done using rules. Riffle is just there to add white space to aesthetics.

Lastly, we have the rules, and some helper functions:

$cellRules = {
   {body_, "Code"} :> boxesToStrings[body],
   {body_String, "Text"} :> commentate[body]};

The contents of the "Code" cells are going to be boxes, and I turn the text in text cells into comments:

$scratchContext = "Scratch`Private`";

Attributes[stringulate] = {HoldAllComplete};
stringulate[expr_] :=
 ToString[Unevaluated[expr], InputForm, CharacterEncoding -> "ASCII", 
 PageWidth -> 72]

boxesToStrings[boxes_] :=
 Block[{$Context = $scratchContext},
  stringulate @@ 
   ToExpression[StripBoxes[boxes], StandardForm, HoldComplete]]

There's a little extra ceremony to keep this function from interning symbols into random contexts, and I had to introduce an extra function, stringulate, to keep everything held until I make it into a string. The handling for text is simpler:

commentate[s_String] :=
 ToString["(*\n" <> s <> "\n*)", OutputForm, 
  CharacterEncoding -> "ASCII", PageWidth -> Infinity]

Ultimately, I'll probably add some rules for handling "Title" cells, "Section"/"Subsection"/... cells, and the like. In particular, if you have a pair of comments like this:

(* ::Section:: *)    
(* The section title *)

the code editor in Wolfram Workbench will fold the thing up like a notebook section when you double click on it. In the end, I'll probably also want to hold onto the whole Cell expressions so that I can pick out input cells that are also initialization cells, because those should be dumped to the package as well.

Answer 3

Here is a basic converter. It depends on the input notebook consisting of input cells only; no output or text cells, etc.

convert[notebook_, package_] := Module[{nb, c, data, d2, held, str},
  nb = NotebookOpen[notebook, Visible -> False];
  SelectionMove[nb, All, Notebook];
  c = NotebookRead[nb];
  NotebookClose[nb];
  If[Head[c] =!= List, c = {c}];
  (* Take the box data out of the cells *)
  data = First /@ c;
  (* Split compound expressions into separate BoxData elements *)
  d2 = Flatten[
    If[Length[First@#] > 1, Map[BoxData, First@#], #] & /@ data, 1];
  held = FullForm[ToExpression[#, TraditionalForm, Hold]] & /@ d2;
  str = StringTake[ToString[# /. "\"" -> "\\\""], {6, -2}] & /@ held;
  Export["text.txt", DeleteCases[ str, "Null"]];
  Quiet[DeleteFile[package]];
  RenameFile["text.txt", package];]

convert["notebook.nb", "package.m"]

<< package.m

UPDATE

Here is an improved version using ExportPacket mentioned here. The package output is now much more readable.

convert[notebook_, package_] := Module[{nb, str},
  nb = NotebookOpen[notebook, Visible -> False];
  SelectionMove[nb, All, Notebook];
  str = First[
    FrontEndExecute[
     FrontEnd`ExportPacket[NotebookSelection[nb], "InputText"]]];
  NotebookClose[nb];
  Export["text.txt", str];
  Quiet[DeleteFile[package]];
  RenameFile["text.txt", package];]

convert["notebook.nb", "package.m"]

<< package.m

This convert function could be run in a front-end via the command line, but a question remains of how to strip out text cells and output, and other non-input code cells from the input notebook.