How to search a directory of nb files for a string?

Question

I would like to find a particular string in a set of nb files in a directory but am having problems interpreting the results of my code. Here's a simple example:

In directory d:\mathematica, I have the following 6 nb files:

theFileNames = {"benchmarks.nb", "ILE stylesheet.nb", 
  "iteration121420.nb", "iteration121720.nb", "testExample.nb", 
  "vectorIterationPlot.nb"}

In testExample.nb, I inserted the string "this is my string". Next I would like to search all the nb files in d:\mathematica and locate any that have this string:

searchData = Table[
   theName = "d:\\mathematica\\" <> theFileNames[[i]];
   nb = NotebookOpen[theName, Visible -> False];
   result = NotebookFind[nb, "this is my string"];
          NotebookClose[nb];
   {i, result},
   {i, 1, Length@theFileNames}
   ];
searchData

Which returns:

{{1, $Failed}, {2, $Failed}, {3, $Failed}, {4, $Failed}, {5, 
  NotebookSelection[
   NotebookObject[FrontEndObject[LinkObject["g5uyb_shm", 3, 1]], 
    122]]}, {6, $Failed}} 

Showing the fifth file returned a match. I would like to identify this file name but I can't simply code:

Position[searchData,{x_,y_}/;y!="$Failed"]

or:

 Position[searchData,{x_,y_}/;y!=$Failed]

as neither of these seem to work. However this:

Position[searchData, {x_, y_} /; y == $Failed]

appears to find all the failed ones but is a bit messy and requires further processing. Can someone explain how to better identify the matching file names or suggest a more efficient way of searching a directory of nb files for a particular string?

Answer 1

This method is fast, but it might not find the string every time. Basically, it reads the notebook file like a text file and looks for the search string. It depends on Mathematica storing the string as a sequence of adjacent characters in the notebook file, which may not be true in every case. If the string is stored as words, surrounding by quote marks and separated by commas, this method would fail.

The reason this method is fast is that it reads the file as a stream.

The method was tested on directory containing 245 notebooks. One of the notebooks is over 400MB, but most are less than 1MB. First, we make a list of all the files in the directory. Your directory and results will be different.

fnames = FileNames["*.nb", "/home/louis/Documents/Wolfram Mathematica"];
Length[fnames]   (*  245  *)
UnitConvert[Total[FileSize /@ fnames], "Megabytes"]   (*  572.568 MB  *)

Now use Table to loop over the file names and use Find to locate the search string. Tally counts the number of times the string is not found and AbsoluteTiming shows how many seconds elapsed. The search string is the text of an error message. The actual message contains "Numerical", capitalized, so we use IgnoreCase->True.

searchString = "numerical integration converging too slowly";

Table[stream = OpenRead[fn];
   result = Find[stream, searchString, IgnoreCase -> True];
   Close[stream];
   If[TrueQ[result == EndOfFile], result, fn],
   {fn, fnames}] // Tally // AbsoluteTiming

    (*  {0.260562, {{EndOfFile, 239}, 
    {"/home/louis/Documents/Wolfram Mathematica/distributions.nb",    1}, 
    {"/home/louis/Documents/Wolfram Mathematica/Hartree-Fock_demo_01.nb",
 1},
    {"/home/louis/Documents/Wolfram Mathematica/helium3.nb",    1},
    {"/home/louis/Documents/Wolfram Mathematica/LCAO-01.nb",    1}, 
    {"/home/louis/Documents/Wolfram Mathematica/magnet.nb",    1}, 
    {"/home/louis/Documents/Wolfram Mathematica/overlap.nb",    1}}}  *)

The output shows the entire loop took about 1/4 second to execute. Find returned EndOfFile 239 times and found the search string (ignoring case) in 6 files.

Note that Find only returns the first occurrence of the search string, if it is found. Use FindList to find all occurrences. When the search string is not found, FindList returns an empty list instead of EndOfFile.

Answer 2

Here's a routine using LouisB's code to search an entire disc drive for nb files containing a selected string. I seeded three files with "This is a test" and the routine running in parallel on a 4.5 GHz quad core machine searched through 60 Gb of 1518 nb files in 78 seconds and found them:

searchString = "This is a test";
(*  retrieve a list of all directories on d-drive*)
dList = Select[FileNames["*", "d:\\", \[Infinity]], DirectoryQ];
Print[Style[dList // MatrixForm, 10]]
(*  search all directories for *.nb files*)
fnames = FileNames["*.nb", #] & /@ dList;
nbFileTotals = Length[#] & /@ fnames   
(* pick out directories with nb files*)
nbDirIndex = Position[nbFileTotals, x_ /; x != 0] // Flatten
nbDirList = dList[[nbDirIndex]];
nbDirList // MatrixForm
(* do the search *)
AbsoluteTiming[
 dTable = ParallelTable[
    currentDir = nbDirList[[dNum]];
    Print["Analyzing ", currentDir];
    fnames = FileNames["*.nb", currentDir];
    totalFiles = Length[fnames] ;  
    fileSizes = FileSize /@ fnames;
    totalSize = Plus @@ (QuantityMagnitude@# & /@ fileSizes);
    searchTable = Table[
      stream = OpenRead[nb];
      result = Find[stream, searchString, IgnoreCase -> True];
      Close[stream];
      If[TrueQ[result == EndOfFile], result, nb],
      {nb, fnames}];
    (*{currentDir,totalFiles,totalSize,searchTable//Tally},*)
    {dNum, totalFiles, totalSize, searchTable // Tally},
    {dNum, 1, Length@nbDirList}];
 ]
dTable // MatrixForm

foundRec = 
 Position[Length@# & /@ dTable[[All, 4]], x_ /; x > 1] // Flatten
If[Length@foundRec > 0,
  foundFileRecords = dTable[[foundRec, 4, 2 ;;]];
  Print[(#[[1, 1]] & /@ foundFileRecords) // MatrixForm];
  ,
  Print["No files found with string: ", searchString];
  ];
(*(d:\Buchberger\buchBerger041021.nb
d:\FilesFromLaptop\USB Drive\Region2CaseA092720.nb
d:\mathematica\stringSave2.nb

)*)