Add package markdown to autogenerated package

Question

I develop in .nb files and autogenerate my packages from them (by setting the notebook option AutoGeneratedPackage -> Automatic) and have no desire to change this, as with the right style settings (crucially InitializationGroup) and some block-indentation code it's very comfortable and easy to toggle from testing to development.

Unfortunately one issue is that I lose the nice package markdown that comes from editing in .m files. I'm able to strip the ugly package header by messing with FrontEndResourceString["PackageHeader"], but I don't have a way to reinstate the markdown.

To see the markdown, edit the .m file using Mathematica by adding Section/Subsection etc. cells. Then, view the saved .m file in a regular text editor.

Is there a way?

I know I can do something hacky like generate it myself via ExportAutoReplacements, e.x. set

{
  "\[LeftDoubleBracket]"->"(* ::Section:: *)\n(*",
  "\[RightDoubleBracket]"->"*)"
 }

or whatever, but this is just so much more unwieldy than whatever internal thing they must use to generate the markdown.

Answer 1

Here is a fairly long answer describing a method to achieve your goals using a stylesheet and a custom TemplateBox.

TemplateBox

The following is a TemplateBox that exports the way you want. For example:

TemplateBox[
    {"section heading", "Section"},
    "Markup",
    DisplayFunction -> (#&),
    InterpretationFunction -> (
        Cell@TextData[{"(* ::", #2, ":: *)\n(*", Cell[BoxData[#]], "*)"}]&
    )
]

This TemplateBox will display as section heading, but will copy as (* ::Section:: *)\n(*section heading*) when a notebook containing this TemplateBox autogenerates a package file. For example, evaluating the following:

CreateDocument[
    {
    Cell[
        BoxData @ TemplateBox[
            {"section heading", "Section"},
            "Markup",
            DisplayFunction -> (#&),
            InterpretationFunction -> (
                Cell@TextData[{"(* ::", #2, ":: *)\n(*", Cell[BoxData[#]], "*)"}]&
            )
        ],
        "Section",
        Evaluatable->True,
        System`InitializationGroup -> True
    ]
    },
    AutoGeneratedPackage -> Automatic
];  

and then saving the resulting notebook will create an autogenerated package with a Section cell.

Section Appearance

Next, we need to make the output look more like it is supposed to (font/color), and we need to have some automated way of using the TemplateBox. The following enhancement to the TemplateBox takes care of font/color:

TemplateBox[
    {"section heading", "Section"},
    "Markup",
    DisplayFunction -> (StyleBox[#,
        FontFamily -> CurrentValue[{StyleDefinitions, #2, FontFamily}],
        FontColor -> CurrentValue[{StyleDefinitions, #2, FontColor}]
    ]&),
    InterpretationFunction -> (
        Cell@TextData[{"(* ::", #2, ":: *)\n(*", Cell[BoxData[#]], "*)"}]&
    )
]

Mangled TemplateBox Issue

One final comment about this TemplateBox. If the TemplateBox is at the top level, then the markup in the InterpretationFunction will be correctly exported to the autogenerated package. However, if the TemplateBox is not at the top level, then things will go wrong. For example, a structure like:

Cell[
    BoxData @ RowBox[{
        TemplateBox[{"stuff", "Section"}, "Markup"],
        "more stuff"
    }],
    "Section"
]

where I've removed some extraneous stuff, will produce an empty Section cell, and a code cell with (*stuff*)more stuff.

For this reason, it will be very useful if the ExportedSection cell has a visible marker for content outside of the TemplateBox.

StyleKeyMapping

Now, I'm going to show a StyleKeyMapping approach to automating the use of the TemplateBox. An example of a StyleKeyMapping can be found in the Input style:

CurrentValue[{StyleDefinitions, "Input", StyleKeyMapping}]

{"Tab" -> "Code", "=" -> "WolframAlphaShort", "*" -> "Item"}

When the insertion point in an Input cell is placed at the very beginning of the cell, entering TAB will convert the Input cell into a Code cell. Similarly, entering = will convert to a WolframAlpha query cell, and * will convert to an Item cell.

ExportedSection Style

For my purposes, in a Section cell, I want TAB to convert the cell into an ExportedSection cell, that has the right properties for a cell to be included in the autogenerated package. Here is an example of an ExportedSection cell style:

Cell[StyleData["ExportedSection", StyleDefinitions->StyleData["Section"]],
    StyleKeyMapping->{"Tab"->"FromExport"},
    Evaluatable->True,
    System`InitializationGroup->True,
    Background->GrayLevel[.95],
    FontColor->RGBColor[0,.5,.0]
]

Some explanation:

1. StyleDefinitions -> StyleData["Section"] will inherit styles from the Section style.

2. I've included a StyleKeyMapping to revert to a normal Section style. I will explain later why I don't just use "[Tab](http://reference.wolfram.com/language/ref/Tab)" -> "Section".

3. Evaluatable->True is needed so that the cell is included in the autogenerated package.

4. InitializationGroup->True is also needed so that the cell is included in the autogenerated package. The reason I use System`InitializationGroup is because the InitializationGroup symbol doesn't exist in a fresh kernel.

5. Background->GrayLevel[.95] is used to indicate that the cell will be included in the autogenerated package.

6. Earlier I discussed how things will go wrong if there is content outside of the TemplateBox. By giving the ExportedSection style a font color of green, content outside the TemplateBox will be green, while content inside the TemplateBox is controlled by the FontColor of the DisplayFunction. This makes it easy to see when the ExportedSection style is mangled and will not export well.

CellDynamicExpression

Now, we are finally ready to discuss how to insert the TemplateBox (at the top level) into an ExportedSection cell. There is a Cell option, CellDynamicExpression:

WolframLanguageData["CellDynamicExpression", "PlaintextUsage"]

"CellDynamicExpression is an option for cells that specifies an expression to \ be dynamically updated whenever the cell is visible on screen."

We can use CellDynamicExpression in the style that is created when the StyleKeyMapping takes place. I will use the RHS of the CellDynamicExpression to wrap the contents of the cell in the TemplateBox.

Now, I'm very concerned about having CellDynamicExpression active in a cell, because a continually triggering CellDynamicExpression will have unwanted effects. So, I will not have the Section style map to ExportedSection directly. Instead, I will map it to a ToExportedSection style, which will then use CellDynamicExpression to create a new cell with style ExportedSection and with the TemplateBox in the right place.

By the way, I really wanted to have just a ToExport style that automatically creates the right kind of cell when used as the target in StyleKeyMapping, but unfortunately, the new cell that is created has no indication of the previous style of the cell. So, I will need to create a ToExportedXXX style for each kind of cell style that I want to have exported.

ToExportedSection

Here is a prototype of the ToExportedSection style:

Cell[StyleData["ToExportedSection",StyleDefinitions->StyleData["Section"]],
    CellDynamicExpression :> With[
        {
        content = func[NotebookRead[EvaluationCell[]]]
        },

        NotebookWrite[
            EvaluationCell[],
            content,
            All
        ];
        SelectionMove[EvaluationNotebook[],After,CellContents];
        SelectionMove[EvaluationNotebook[],Previous,Character]
    ],
    DefaultFormatType->InputForm
];

The function func takes the cell contents, and creates a Section cell with the right structure. Then NotebookWrite replaces the old cell with this new cell. Finally, I move the insertion point to just the right place so that further typing will just add on to the content in the TemplateBox.

DefaultFormatType

The DefaultFormatType for a Section cell is DefaultTextFormatType. Since I am creating a box structure from the contents of the Section cell I want to use DefaultInputFormatType instead. If I use the default, then a Section cell with inline content could get mangled. With DefaultInputFormatType the contents will just be an appropriate box representation of the original Section cell.

Section

The ToExportedSection style is what I want to point to in the StyleKeyMapping for "Section":

Cell[StyleData["Section"],
    StyleKeyMapping->{"Tab" -> "ToExportedSection"}
]

FromExport

Earlier I mentioned that the StyleKeyMapping for ExportedSection pointed to FromExport instead of Section. This is because I want to remove the TemplateBox when converting back, and so just like I needed a ToExportedSection style to add the TemplateBox, I need a FromExport style to strip the TemplateBox. However, in this case, the cell that is obtained by NotebookRead @ EvaluationCell[] has the necessary markup information to create the right kind of cell to convert to. Hence, I don't need a FromExport style for each style that I'm going to export.

Usage

Those are most of the issues in creating a stylesheet to enable simple creation of exportable styles, and the full stylesheet is given at the end of this answer. This stylesheet include exportable Section, Subsection and Text styles.

The process of creating content using this style sheet is as simple as using the keyboard shortcut to create the style (e.g., ALT-4 to create the Section style), and then hit Tab to convert to the ExportedSection style, and then type your section name. Similarly for the Subsection (ALT-5) and Text (ALT-7) styles.

If you later want to edit the cell, you can just type as normal. If you happen to not be in the TemplateBox, you will find that your added content will be green and not the correct style color. In this case, just erase and move the cursor so that it is in the TemplateBox and try again. Here is a screenshot where text is added outside of the TemplateBox:

Hopefully the green color is a sufficient clue that text is not being added to the TemplateBox.

Stylesheet

Below is the stylesheet, given as a Notebook expression assigned to the variable nb. You can either use:

SetOptions[EvaluationNotebook[], StyleDefinitions->nb]

in the notebook that you want to autogenerate a package from (remember to also use SetOptions[EvaluationNotebook[], AutogeneratedPackage->Automatic] as well), or you can install the style sheet using:

NotebookPut @ nb

and then select Install Stylesheet ... and fill in the dialog to give the stylesheet a name and install it into your system. Then you would just select this stylesheet using the Format | Stylesheet menu item.

nb = Notebook[{
    Cell[StyleData[StyleDefinitions->"Default.nb"]],
    Cell[StyleData["Markup"],
        TemplateBoxOptions->{
            DisplayFunction->(StyleBox[
                #1,
                FontColor->CurrentValue[{StyleDefinitions,#2,FontColor}],
                FontFamily->CurrentValue[{StyleDefinitions,#2,FontFamily}]]&
            ),
            InterpretationFunction->(
                Cell[
                    TextData[{"(* ::", #2, ":: *)\n(*", Cell[BoxData[#]], "*)\n"}]
                ]&
            )
        }
    ],
    Cell[StyleData["Section"],
        StyleKeyMapping->{"Tab"->"ToExportedSection"}
    ],
    Cell[StyleData["ExportedSection", StyleDefinitions->StyleData["Section"]],
        StyleKeyMapping->{"Tab"->"FromExport"},
        Evaluatable->True,
        System`InitializationGroup->True,
        Background->GrayLevel[.95],
        FontColor->RGBColor[0,.5,.0] (* to indicate text outside of template *)
    ],
    Cell[StyleData["ToExportedSection",StyleDefinitions->StyleData["Section"]],
        CellDynamicExpression :> With[
            {
            content = Replace[
                First @ NotebookRead[EvaluationCell[]],
                BoxData[x_] :> x
            ]
            },

            NotebookWrite[
                EvaluationCell[],
                Cell[
                    BoxData @ TemplateBox[{content, "Section"}, "Markup"],
                    "ExportedSection"
                ],
                All
            ];
            SelectionMove[EvaluationNotebook[],After,CellContents];
            SelectionMove[EvaluationNotebook[],Previous,Character]
        ],
        DefaultFormatType->DefaultInputFormatType
    ],
    Cell[StyleData["Subsection"],
        StyleKeyMapping->{"Tab"->"ToExportedSubsection"}
    ],
    Cell[StyleData["ExportedSubsection", StyleDefinitions->StyleData["Subsection"]],
        StyleKeyMapping->{"Tab"->"FromExport"},
        Evaluatable->True,
        System`InitializationGroup->True,
        Background->GrayLevel[.95],
        FontColor->RGBColor[0,.5,.0] (* to indicate text outside of template *)
    ],
    Cell[StyleData["ToExportedSubsection",StyleDefinitions->StyleData["Subsection"]],
        CellDynamicExpression :> With[
            {
            content = Replace[
                First @ NotebookRead[EvaluationCell[]],
                BoxData[x_] :> x
            ]
            },

            NotebookWrite[
                EvaluationCell[],
                Cell[
                    BoxData @ TemplateBox[{content, "Subsection"}, "Markup"],
                    "ExportedSubsection"
                ],
                All
            ];
            SelectionMove[EvaluationNotebook[],After,CellContents];
            SelectionMove[EvaluationNotebook[],Previous,Character]
        ],
        DefaultFormatType->DefaultInputFormatType
    ],
    Cell[StyleData["Text"],
        StyleKeyMapping->{"Tab"->"ToExportedText"}
    ],
    Cell[StyleData["ExportedText", StyleDefinitions->StyleData["Text"]],
        StyleKeyMapping->{"Tab"->"FromExport"},
        Evaluatable->True,
        System`InitializationGroup->True,
        Background->GrayLevel[.95],
        FontColor->RGBColor[0,.5,.0] (* to indicate text outside of template *)
    ],
    Cell[StyleData["ToExportedText",StyleDefinitions->StyleData["Text"]],
        CellDynamicExpression :> With[
            {
            content = Replace[
                First @ NotebookRead[EvaluationCell[]],
                BoxData[x_] :> x
            ]
            },

            NotebookWrite[
                EvaluationCell[],
                Cell[
                    BoxData @ TemplateBox[{content, "Text"}, "Markup"],
                    "ExportedText"
                ],
                All
            ];
            SelectionMove[EvaluationNotebook[],After,CellContents];
            SelectionMove[EvaluationNotebook[],Previous,Character]
        ],
        DefaultFormatType->DefaultInputFormatType
    ],
    Cell[StyleData["FromExport",StyleDefinitions->StyleData["Section"]],
        CellDynamicExpression :> With[
            {
            cell = Cases[
                First @ NotebookRead @ EvaluationCell[],
                TemplateBox[{a_, b_}, "Markup"] :> Cell[a, b],
                Infinity,
                1
            ]
            },

            NotebookWrite[
                EvaluationCell[],
                Replace[
                    cell,
                    {
                    {Cell[a_String, b_]} :> Cell[a, b],
                    {Cell[a_, b_]} :> Cell[BoxData[a], b]
                    }
                ],
                All
            ];
            SelectionMove[EvaluationNotebook[],After,CellContents]
        ],
        DefaultFormatType->DefaultInputFormatType
    ]
    },
    StyleDefinitions->"PrivateStylesheetFormatting.nb"
];

Answer 2

I'm still hoping for a proper answer but here's the work-around I've been working with.

Since I use IninitailizationGroup my "Section", "Subsection", etc. cells are already tagged as being InitializationCells so I don't have to do anything there. They do, however, need to be made Evaluatable to be exported to the package file.

They'll be exported as raw code, however, unless we hack the markdown in.

So what I did was set up a standard left and right character that would format the markdown and for each style I set

ExportAutoReplacements->{
 leftEscape->"(* ::"<>style<>":: *)\n(*",
 rightEscape->"*)"
 }

and

CellEventActions->{

{"MenuCommand","HandleShiftReturn"}:>
 With[{
   ev = CurrentValue[EvaluationCell[], Evaluatable],
   text = First@NotebookRead[EvaluationCell[]]
   },
  SelectionMove[EvaluationCell[],
   All,
   CellContents
   ];
  NotebookWrite[EvaluationNotebook[],
   If[ev // TrueQ,
    StringTrim[text,
     (StartOfString ~~ (Whitespace | "") ~~ leftEscape) |
      (rightEscape ~~ (Whitespace | "") ~~ EndOfString)
     ],
    leftEscape <> text <> rightEscape
    ]
   ];
  SetOptions[EvaluationCell[], {
    Evaluatable -> Not@ev
    }];
  ]

}

This way I can easily toggle which markdown cells get exported and make sure they have the appropriate formatting.

Then there's the question of what to use for your leftEscape and rightEscape and unfortunately here I think there's no good choice. They obviously can't be the same and if you pass a multiple character string to ExportAutoReplacements it will treat each individual character as an instance for replacement which is not the desired behavior.

In the end I went with leftEscape="\[LeftCeiling]" and rightEscape="\[RightFloor]" because it's not too aesthetically displeasing and I doubt I will ever actually want to use either of them in my cell text.

And this works fine, even if it is hacky and unsatisfying. As an example here's the side-by-side comparison of my dev notebook and the formatted package: