10
$\begingroup$

I have one single notebook with more than 500 mb...

To control its growth, I would like to have a profiler of used space (not the live memory, but the space on disk), in such a way that I can easily pinpoint the cells most responsible for its size (sometimes, its just a matter of playing with resolution and magnification).

I imagine a function that generates a notebook index, starting by the biggest spenders, and going down, on which I can click and jump to the guilty ones.

I'm not very proficient at this kind of notebook level programming, and hence, the request for help...

$\endgroup$
0

2 Answers 2

9
$\begingroup$

Here is another try, this time more close to what you actually wanted: It generates a list of buttons for the largest cells (button is labeled with size in kB). If you press a button, the corresponding cell will be selected. You can change the number of rows to show if you wish.

The code needs the CellIDs to be set, so it has buttons to create and delete such ids (there is not a real reason to ever delete them, but if the notebook has not had set CreateCellID, then they won't exist). I have not tested the generation of CellIDs with a 500MB notebook, what I do to get them generated is somewhat naive, so be sure to only try the buttons to create/delete cellids with a copy of your notebook (the other code should not do any harm even when it crashes...). If you run into problems, you can of course create CellIDs manually as well: Set CreateCellID->True for your notebook and do a cut and paste per cell (or cellgroup or section or whatever works well). More work but probably safer...

Here is the code for the palette:

CreatePalette[
  DynamicModule[{
      numrows=5,update,buttons={},numcells, cellcount
    },
    update=Function[nb,
      Module[{cellcontent,cellsizes={},thisid,prgrsdia = Null, progress = 0},
        cellcount = 0;
        SelectionMove[nb, All, Notebook];
        numcells = CurrentValue[nb, "CellCount"];
        If[numcells > 10,
          prgrsdia = CreateDialog[
            ProgressIndicator[Dynamic[N[cellcount/numcells]]],
            WindowTitle -> "Cell-Size-Progress"
          ]
        ];
        SelectionMove[nb,Before,Notebook];
        SelectionMove[nb,Next,Cell];
        cellcontent=NotebookRead[nb];
        While[cellcontent=!={},
          thisid=CurrentValue[NotebookSelection[nb],CellID];
          If[thisid==0,
            MessageDialog["found cell with not tag, aborting..."];
            Abort[]
          ];
          cellsizes={cellsizes,thisid->ByteCount@cellcontent};
          SelectionMove[nb,Next,Cell,AutoScroll->False];
          cellcontent=NotebookRead[nb];
          cellcount++;
        ];
        If[prgrsdia =!= Null, NotebookClose[prgrsdia]];
        Apply[
          Button[#2,NotebookFind[nb,#1,All,CellID],ImageSize->Scaled[1]]&,
          Reverse@SortBy[Flatten[cellsizes],Last],
          {1}
        ]
      ]
    ];
    Column[{
      Button["Create CellIDs",
        With[{nb=InputNotebook[]},
          SetOptions[nb,CreateCellID->True];
          SelectionMove[nb,All,Notebook];
          FrontEndTokenExecute[nb,"Cut"];
          FrontEndTokenExecute[nb,"Paste"];
        ],
        Method->"Queued"
      ],
      Button["Remove CellIDs",
        With[{nb=InputNotebook[]},
          SetOptions[nb,CreateCellID->False];
          NotebookPut[
            NotebookGet[nb] /. {
              Cell[x___,Verbatim[Rule][CellID,_],y___]:>Cell[x,y]
            },
            nb
          ];
        ],
        Method->"Queued"
      ],
      Row[{
        Button["Update",buttons=update[InputNotebook[]],Method->"Queued"],
        Button["+",numrows++],
        Button["-",numrows--]
      }],
      Dynamic[Column[Take[buttons,Clip[numrows,{1,Length[buttons]}]]]]
    }]
  ],
  WindowTitle->"Large Cells"
];

EDIT: I have added now a progress bar which will show the progress when updating the list of cell sizes. Note that the code which adds and removes the cell tags does copy the complete content of the notebook into kernel memory, which might fail for huge notebooks. If that would be the case, one could use an approach similar to the one which determines the cell sizes and add/remove tags cell by cell. While that would be more memory efficient, it might be much slower. To get a good balance one probably would need to use a chunked approach: select e.g. 10 cells and treat them in one go, then select and treat the next 10 cells etc. I have not tried if and how much that will speed things up and don't have time to do so...

$\endgroup$
4
  • $\begingroup$ It takes its time, but worked great on a 600 MB file. $\endgroup$
    – P. Fonseca
    Dec 17, 2015 at 4:29
  • $\begingroup$ @P.Fonseca: good to hear. Do you remember what exactly was taking long? I would expect the creation of cell ids to last long - it is somewhat ad hoc. But that would only have to be done once per notebook, if you keep the CreateCellID set to True for that notebook, new cells will also get a unique id. The generation of the list could probably be made faster by reading the whole notebook at once, but that will take some effort and probably become a memory problem as we then would load the whole notebook content into kernel memory... $\endgroup$ Dec 17, 2015 at 10:40
  • $\begingroup$ I think that both tasks took approximately the same time. Completely acceptable for the purpose of the tool (like 1 min. max.), but since you don't know if it is working or not, it just feels like a lot of time. I don't know if there's a method to know how many cells a notebook has. If there is, a progress bar would probably be interesting. Also interesting is the fact that the big spender cells, are definitely not what I would have guessed... and so, this is a very practical tool! I have several thousand cells, and yet, just one was taking 15% of the total size! $\endgroup$
    – P. Fonseca
    Dec 17, 2015 at 12:01
  • $\begingroup$ @P.Fonseca: a progress bar would certainly make sense. Difficulty might be to count the total number of cells without reading the complete Notebook expression into kernel memory. I might add that when I have more time to look at this. Good to hear it was helpful, and yes, I think 1 minute is probably OK for the purpose it serves and such large notebooks... $\endgroup$ Dec 17, 2015 at 13:28
3
$\begingroup$

The code below does not exactly what you have asked for, but it should contain the relevant stuff. To create an index ordered by cell sizes I think it would be easiest to set CreateCellID to True for your notebook. To create ids for the existing cells you'd have to "Cut" and "Paste" all of them once, new cells will automatically get an unique CellID. Then you could use code similar to the one below to collect a list of cell-ids and corresponding cell-sizes, order that list by cell-size and write a column of buttons which do a NotebookFind or NotebookLocate to select the given cell-id in your notebook.

The code following creates a palette which will add a cell-tag showing the size of each cell in Kilobytes. There is also a button which removes those cell tags and one which switches the "ShowCellTags" back to its default False:

CreatePalette[Column[{
   Button["Show Sizes",
    Module[{nb = InputNotebook[], cellcontent},
     SelectionMove[nb, Before, Notebook];
     SelectionMove[nb, Next, Cell];
     cellcontent = NotebookRead[nb];
     While[cellcontent =!= {},
      SetOptions[NotebookSelection[nb], CellTags -> Append[
         Select[
          Flatten[{CurrentValue[NotebookSelection[nb], CellTags]}],
          Not@StringMatchQ[#, "size:" ~~ DigitCharacter .. ~~ "kB"] &
          ],
         "size:" <> ToString[Round[ByteCount@cellcontent/10.^3]] <> 
          "kB"
         ]
       ];
      SelectionMove[nb, Next, Cell];
      cellcontent = NotebookRead[nb];
      ]
     ],
    Method -> "Queued"
    ],
   Button["Remove Size Tags",
    Module[{nb = InputNotebook[], cellcontent},
     SelectionMove[nb, Before, Notebook];
     SelectionMove[nb, Next, Cell];
     cellcontent = NotebookRead[nb];
     While[cellcontent =!= {},
      SetOptions[NotebookSelection[nb],
       CellTags -> Select[
         Flatten[{CurrentValue[NotebookSelection[nb], CellTags]}],
         Not@StringMatchQ[#, "size:" ~~ DigitCharacter .. ~~ "kB"] &
         ]
       ];
      SelectionMove[nb, Next, Cell];
      cellcontent = NotebookRead[nb];
      ]
     ],
    Method -> "Queued"
    ],
    Button["Hide Cell-Tags",
      CurrentValue[nb, ShowCellTags] = False,
      Method -> "Queued"
    ]
}], WindowTitle -> "Cell Sizes"]
$\endgroup$
1
  • $\begingroup$ +1 This is indeed a good start. Thank you. $\endgroup$
    – P. Fonseca
    Jun 29, 2015 at 11:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.