# Programmatically generate packages from notebook files?

Currently, there are two ways I know of to generate a package (.m) file from a notebook file; one is to set the AutoGeneratedPackage option in the Option Inspector, and the other is to use the menu optoin to "Save as" a package. The first doesn't really suit my needs because it creates the auto-generated packages in the same directory as the notebook file, which clutters things up (there are dozens of notebooks in this directory) and adds a needless complication to keeping the set-up version controlled. The second option is less than ideal because I don't know how to invoke the option programmatically.

In theory, I know it might be possible with FrontEnd commands, but I'd really prefer to be able to do this without a front end if at all possible. This is something I want to do from GNU make, and while I know you can now set up front ends from the command line, ideally I could avoid that.

-
I do a programmatic package generation from held code in the code for the syntax highlighter, described here. This can be one starting point. You can grab the code from a gven notebook using Cases, in the box form, and then convert to held expressions. I did play with the code a bit, so could post it if you think this could be an option for you. –  Leonid Shifrin Feb 23 '12 at 18:43
@LeonidShifrin That's a little more work than I was hoping to have to do, but it's a promising approach. I actually have some existing code (which I use to generate other documents from the notebooks) which does that sort of thing as part of the build process. –  Pillsy Feb 23 '12 at 19:43
Simple suggestion: why not just keep the source .nb in the destination for the .m package, then save the notebook as needed after -- just once -- setting the AutoGeneratedPackage option for it? –  murray Feb 23 '12 at 22:27

Not entirely on topic (and probably well-known), but starting with version 6 (I think) you can very comfortably work with the .m files within the frontend. The frontend parses the package and displays it just like any old notebook. You can use sections and do all the fancy typesetting, generate (temporary) output etc. This is also available under File -> New -> Package (.m).

The code is put into "Code" style cells which are Initialization cells, and these are saved (together with text and section information, see Pillsy´s post), stripping any output cells and leaving you with the plain package that works just fine. With the parsing, you also get some helpful formatting and additional gizmos (see image).

So you only maintain one set of packages directly, but with all the benefits of the frontend interface (notice: the parsing can take a few seconds when opening).

And if you are annoyed by the linewrapping behaviour of the package stylesheet, have a look at this thread: Viewing packages in Mathematica

-

I tried the ExportPacket approach described in Chris Degnen's answer, and while it's very simple, the need for FrontEndExecute and the difficulties dealing with cells of text and section headings and the like make it less appealing. Here's a sketch of the approach I'll ultimately be using, though I haven't filled all the details in yet:

notebookToPackage[notebook_String, package_String] :=
With[{nbObj = Get@notebook},
If[
nbObj === $Failed,$Failed,
Export[package, notebookToStrings@nbObj, {"Text", "Lines"}]]]


This just loads the notebook file and does some basic error handling and exports the result after notebookToStrings converts the mess into strings. Using Get to get the notebook is the easiest way to get it as an object with head Notebook, and from there you can get at the Cell structures.

notebookToStrings[nbObj_] :=
With[{cells =
Cases[nbObj,
Cell[body_, style : "Code" | "Text", ___] :> {body, style},
Infinity]},
Riffle[
Replace[cells,
$cellRules, {1}], ""]]  Since notebook objects are Mathematica structures like everything else, you can use the same old structural manipulations and rule replacements that you use everywhere else. Cases proves once again that, with the Infinity level specification, it's the most useful function in the Mathematica library, and the pattern matching means you can whitelist some cells while not having to decide what you want to do about input or output cells or subsections or whatever. Everything else is done using rules. Riffle is just there to add white space to aesthetics. Lastly, we have the rules, and some helper functions: $cellRules = {
{body_, "Code"} :> boxesToStrings[body],
{body_String, "Text"} :> commentate[body]};


The contents of the "Code" cells are going to be boxes, and I turn the text in text cells into comments:

$scratchContext = "ScratchPrivate"; Attributes[stringulate] = {HoldAllComplete}; stringulate[expr_] := ToString[Unevaluated[expr], InputForm, CharacterEncoding -> "ASCII", PageWidth -> 72] boxesToStrings[boxes_] := Block[{$Context = \$scratchContext},
stringulate @@
ToExpression[StripBoxes[boxes], StandardForm, HoldComplete]]


There's a little extra ceremony to keep this function from interning symbols into random contexts, and I had to introduce an extra function, stringulate, to keep everything held until I make it into a string. The handling for text is simpler:

commentate[s_String] :=
ToString["(*\n" <> s <> "\n*)", OutputForm,
CharacterEncoding -> "ASCII", PageWidth -> Infinity]


Ultimately, I'll probably add some rules for handling "Title" cells, "Section"/"Subsection"/... cells, and the like. In particular, if you have a pair of comments like this:

(* ::Section:: *)
(* The section title *)


the code editor in Wolfram Workbench will fold the thing up like a notebook section when you double click on it. In the end, I'll probably also want to hold onto the whole Cell expressions so that I can pick out input cells that are also initialization cells, because those should be dumped to the package as well.

-
Your Cases approach fails in certain scenarios because Cases evaluates the right hand side of certain expression types. The issue is described in more detail here –  William Aug 14 '13 at 5:53
@Liam I think that's a potential issue; as of Mathematica 9, we have an actual function called Cells which may do what we want/need without this possible failure mode, though I haven't looked closely at it yet. –  Pillsy Aug 14 '13 at 11:42

Here is a basic converter. It depends on the input notebook consisting of input cells only; no output or text cells, etc.

convert[notebook_, package_] := Module[{nb, c, data, d2, held, str},
nb = NotebookOpen[notebook, Visible -> False];
SelectionMove[nb, All, Notebook];
NotebookClose[nb];
If[Head[c] =!= List, c = {c}];
(* Take the box data out of the cells *)
data = First /@ c;
(* Split compound expressions into separate BoxData elements *)
d2 = Flatten[
If[Length[First@#] > 1, Map[BoxData, First@#], #] & /@ data, 1];
held = FullForm[ToExpression[#, TraditionalForm, Hold]] & /@ d2;
str = StringTake[ToString[# /. "\"" -> "\\\""], {6, -2}] & /@ held;
Export["text.txt", DeleteCases[ str, "Null"]];
Quiet[DeleteFile[package]];
RenameFile["text.txt", package];]

convert["notebook.nb", "package.m"]

<< package.m


UPDATE

Here is an improved version using ExportPacket mentioned here. The package output is now much more readable.

convert[notebook_, package_] := Module[{nb, str},
nb = NotebookOpen[notebook, Visible -> False];
SelectionMove[nb, All, Notebook];
str = First[
FrontEndExecute[
FrontEndExportPacket[NotebookSelection[nb], "InputText"]]];
NotebookClose[nb];
Export["text.txt", str];
Quiet[DeleteFile[package]];
RenameFile["text.txt", package];]

convert["notebook.nb", "package.m"]

<< package.m


This convert` function could be run in a front-end via the command line, but a question remains of how to strip out text cells and output, and other non-input code cells from the input notebook.

-