Automatically counting the number of lines of code in a set of notebooks

Question

How can I automatically count the number of lines of code "Input" existing inside all the notebook files contained in a specified folder?

I know there is the issue of considering or not the automatic line breakings as new lines. But considering the purpose of this exercise (statistically convince that the amount of developed work on Mathematica justifies keeping a license), I'm not searching for exact values.

Answer 1

Assuming that the notebook object is contained in the variable nb (for the current notebook do nb = EvaluationNotebook[]), one could use the following:

StringCount[
 First@FrontEndExecute@FrontEnd`ExportPacket[
    Notebook@Cases[First@NotebookGet[nb], Cell[_, "Input"|"Code", ___]],
    "PlainText"],
 "\n"
 ]

Using "InputText" will wrap lines much more frequently and will give a substantially larger count.

This is equivalent to Alt-clicking an input cell to select all input cells, then copying the contents as "plain text" or "input text" and counting the lines.

As you can see, Mathematica does not really respect line breaks. If you are trying to measure the volume of code, I suggest using the LeafCount of expressions instead of counting code lines.

Answer 2

Here is my version:

ClearAll[nbLines];
 nbLines[nb_NotebookObject] :=
     With[{inputs =
         Cases[NotebookGet[nb], Cell[BoxData[data_], "Input", ___] :> data, Infinity]
       },
      Length[inputs] + 
      Count[Characters@StringJoin@Cases[inputs, _String, Infinity], 
        "\n" | "\[IndentingNewLine]"]
     ]

This may not be precise (because the method of input extraction is not totally robust), but should be good enough for qualitative estimate.

Answer 3

Somehow I do like kernel-only solutions: (* see also the github repository here *)

CountNumberOfLinesOfInputCells::usage = "CountNumberOfLinesOfInputCells[dir] counts the number of lines (with default 
line length 78) of input cells (at top level) of all notebooks in dir. The operation runs completely in a Mathematica kernel, using only MakeExpression and 
some fixes thereof (e.g. input cells are splitted (like Ctrl Shift M) programmatically at newlines, since MakeExpression might not work in such a case.";

Options[CountNumberOfLinesOfInputCells] = {PageWidth -> 78}; 
CountNumberOfLinesOfInputCells[dirs__, OptionsPattern[]] :=
    Module[ {files, n},
        files = FileNames["*.nb", dirs];
        Print["counting lines of  ", 
        Length[files], " files in ", Riffle[Flatten[{dirs}], " "]];
        (*TODO: figure out why ParallelTable does not seem to work here; maybe Streams do not parallelize well? *)
        Total[Monitor[Table[SlocSingleNB[files[[n]], OptionValue[PageWidth]], 
           {n, 1, Length[files]}], n]]
    ]; 
SlocSingleNB[(f_String)?FileExistsQ, pagewidth_:78] :=
    Quiet[Module[ {inputcells, holdcompletelist, o, tmpfile, le},
              inputcells = (Cases[#1, Cell[_BoxData, "Input", ___]] & )[
                 Flatten[Replace[Get[f], Notebook[z_, ___] :> z] //. 
                   Cell[CellGroupData[{c__Cell}, _]] :> c]];
              (* Split inputcells at newlines. This is done because 
              MakeExpression[#,StandardForm]& @ BoxData[{ RowBox[{ RowBox[{"SetAttributes", "[", RowBox[{"f", ",", "Listable"}], "]"}], ";"}], "\n", 
                                                          RowBox[{ RowBox[{"f", "[", "x_", "]"}], ":=", RowBox[{"x", "^", "2"}]}]}] 
              does not work. This cell can be found in Defintion.nb *)
              inputcells = 
              inputcells //. {Cell[BoxData[{r1__, "\n".., r2__}], "Input", ___] :> Sequence[Cell[
              BoxData[{r1}]
              , "Input"], Cell[BoxData[{r2}], "Input"]], Cell[BoxData[{r1__, "\[IndentingNewLine]".., r2__}], 
              "Input"] :> Sequence[Cell[BoxData[{r1}], "Input"], Cell[BoxData[{r2}], "Input"]]};
              (* something is weird with Defition: e.g. 
                  MakeExpression[ RowBox[{"Definition", "[", "f", "]"}], StandardForm]
                  returns HoldComplete[Null]
                  Substitution Definition by Identity[Definition] seems to work.
              *)
              If[ !FreeQ[inputcells, "Definition"],
                  inputcells = inputcells /. 
                  "Definition" -> "Identity[Definition]"
              ];
              holdcompletelist = 
              (MakeExpression[First[#1], StandardForm] //. ErrorBox[err_] :> Sequence[] & ) /@ 
              inputcells;
              o = OpenWrite[];
              Do[WriteString[o, (StringJoin[ToString[#1, InputForm, PageWidth -> pagewidth], 
                   ";\n"] & )[Unevaluated @@ holdcompletelist[[i]]]], {i, Length[holdcompletelist]}];
              tmpfile = Close[o];
              le = Length[Import[tmpfile, "Lines"]];
              DeleteFile[tmpfile];
              le
          ]]; 
 (*
 This takes a couple of seconds and should count 1690 lines of code of 227 notebooks:

  AbsoluteTiming[CountNumberOfLinesOfInputCells[FileNameJoin@{$InstallationDirectory,"AddOns"},Infinity]]

This takes about 10 minutes and counts 13395865 lines of code of 10284 notebooks:
 AbsoluteTiming[CountNumberOfLinesOfInputCells[$InstallationDirectory, Infinity]]
*)

Answer 4

If the notebook contains only input the following rough approach using Import will do:

StringCases[Import["ExampleData/document.nb", "Plaintext"] ,    
  Longest[ {"\n" | "\r"} ..]] // Length

(*
==> 19
*)

If there are output cells with lines of text (graphics output was already removed by the Plaintext import), we can remove the output cells first, export to string as NB and re-import as plain text counting lines of input:

  StringCases[
  ImportString[
   ExportString[
    Import["ExampleData/document.nb"] /. 
     Cell[___, "Output", ___] -> Sequence[],
    "NB"
    ],
   "Plaintext"
   ],    Longest[ {"\n" | "\r"} ..]
  ] // Length

(*
==> 15
*)

Answer 5

Using NotebookImport[] (introduced in 2015, version 10.1) simplifies things a lot:

nbinputlinecount = Length@NotebookImport[notebookpath, "Input"]

where notebookpath is the file path to the desired notebook. This only counts the number of lines of input. However, input lines can have multiple lines within them. To tally the total number of lines within all the input cells use the following:

linesperinput[inputline_] := 
   Length@Select[
       ToString@Extract[inputline, #, Hold] & /@ 
          Range[Depth[inputline] - 1], # != "Hold[Null]" &]

anb = NotebookImport[notebookpath, "Input"]
Total[linesperinput[#] & /@ anb]