> I'm not worried about having to remove stuff manually if I change the cell style from my FigureCaption back to something else. That is a scenario that I would consider rare. So if that is a major concern, I'd say ignore it..

So this should work for you, add those (or analogous) styles to the stylesheet:

    Cell[StyleData["Section"],
     CellDynamicExpression :> (
     	SelectionMove[EvaluationCell[], Before, CellContents];
     	NotebookWrite[
          EvaluationNotebook[], 
          TemplateBox[{CounterBox["Section"]},"TemplatedSection"], 
          After
        ];
     	SetOptions[EvaluationCell[], CellDynamicExpression -> None]     
     ),
     MenuSortingValue->10000]

That part inserts what you want and disables this feature for given cell (because I don't trust `CellDynamicExpression` [126655](http://mathematica.stackexchange.com/q/126655/5478)).

It would be nice to not have to recreate all cells when the "header" is changed so the code above inserts only a `TemplateBox` for `TemplateSelection` which we can define later in the stylesheet and modify separately:

    Cell[
       StyleData["TemplatedSection"],
       TemplateBoxOptions -> {
          DisplayFunction :> (
             TemplateBox[
               {#1,". section note",": "},"RowDefault",Editable->False
             ]&
          )
       },
       MenuSortingValue->10000 
    ]

[![enter image description here][1]][1]

I've noticed that sometime when the `TemplateBox` is edited cells are not updated properly/counter number is broken, but as soon as once creates one section and deletes it, everything is fixed. So that is only a minor inconvenience.


  [1]: https://i.sstatic.net/XBNo1.png