Efficient way to list only files (but not directories) in subdirectories

Question

Currently we have a (not very efficient, but still) function FileNames allowing to get the complete listing of all files and subdirectories of a directory at arbitrary depth. But it has no option allowing to restrict its output to the actual files when directories should be excluded. Hence for selecting only files from its output we are forced to use DirectoryQ, FileType or undocumented FileInformation, all of them don't accept a list as an argument and add significant time to FileNames:

FileNames[All, $InstallationDirectory, Infinity]; // AbsoluteTiming
FileNames[All, $InstallationDirectory, Infinity]; // AbsoluteTiming
l1 = Select[FileNames[All, $InstallationDirectory, Infinity], 
    Not@*DirectoryQ]; // AbsoluteTiming
l2 = Select[FileNames[All, $InstallationDirectory, Infinity], 
    FileType[#] === File &]; // AbsoluteTiming
l3 = Select[FileNames[All, $InstallationDirectory, Infinity], 
    FileInformation[#, "FileType"] === "File" &]; // AbsoluteTiming
l4 = Select[FileNames[All, $InstallationDirectory, Infinity], 
    Information[File@#, "FileType"] === "File" &]; // AbsoluteTiming

l1 == l2 == l3 == l4

{7.1199, Null}

{6.98109, Null}

{12.113, Null}

{22.7185, Null}

{20.8688, Null}

{397.981, Null}

True

Is there a more efficient way to obtain the complete listing of only actual files in a directory and its subdirectories at arbitrary depth?

P.S. There are other methods, but unfortunately they don't work in the latest Methamatica versions:

FileNames[name__ /; ! DirectoryQ[name], $InstallationDirectory, Infinity]

FileNames[StringExpression[
  name__ /; Not[DirectoryQ[name]]], $InstallationDirectory, Infinity]

both return a list containing directories with Mathematica 13.0.0. With version 8.0.4 they work as expected, but slower than the l1 method above.

Answer 1

One way is to use the system dir command. The disadvantage is that it doesn't allow to specify the arbitrary depth (at least on Windows). For Windows:

FileNames[All, $InstallationDirectory, Infinity]; // AbsoluteTiming
l1 = Select[FileNames[All, $InstallationDirectory, Infinity], 
    Not@*DirectoryQ]; // AbsoluteTiming
l5 = ImportString[
    RunProcess[{"cmd", "/c", "dir", "/b", "/s", "/a-d", $InstallationDirectory}, 
     "StandardOutput"], "Lines", 
    CharacterEncoding -> $SystemCharacterEncoding]; // AbsoluteTiming

l1 == Sort@l5    

{7.16807, Null}

{14.5409, Null}

{3.55901, Null}

True

Another possible way which allows to specify arbitrary depth is to use Windows PowerShell as shown in this thread. This answer shows how it can be used from Mathematica.

---

I've tried to reduce the filesystem load by filtering out directory names (found by applying DirectoryName) from the list returned by FileNames. It didn't gave sensible speedup however because in the case of $InstallationDirectory this list contains only about 10% of directories. But when working with network paths it can give appreciable speedup, so I publish here this solution too:

l6 = Module[{l = FileNames[All, $InstallationDirectory, Infinity], NonEmptyDirectories, 
     lFiltered},
    (* StringDrop is needed only under Windows! It drops the ending "\\". *)
    NonEmptyDirectories = StringDrop[DeleteDuplicates[DirectoryName /@ l], -1];
    lFiltered = DeleteCases[l, Alternatives @@ NonEmptyDirectories];
    Select[lFiltered, Not@*DirectoryQ]]; // AbsoluteTiming

l1 == l6

{12.4957, Null}

True

It should be noted that there are 21 empty and 4181 non-empty directories in $InstallationDirectory and 44681 actual files. We are forced to check 44702 filepaths just to remove these 21 empty directories from the list...

Answer 2

If you don't mind installing the free linux emulator Cygwin on your Windows 7 machine and make sure that bash.exe is in your path, then you have the option of running linux commands on all platforms. Here is an example of counting all the files and counting all the files to depth 2.

RunProcess[{"bash", "-c", "/usr/bin/find . -type f | wc -l"},
   ProcessDirectory -> $InstallationDirectory] // AbsoluteTiming
RunProcess[{"bash", "-c", 
   "/usr/bin/find . -maxdepth 2 -type f | wc -l"}, 
  ProcessDirectory -> $InstallationDirectory] // AbsoluteTiming
(* {2.13029, <|"ExitCode" -> 0, "StandardOutput" -> "44678
   ", "StandardError" -> ""|>} *)
(* {0.125235, <|"ExitCode" -> 0, "StandardOutput" -> "11
   ", "StandardError" -> ""|>} *)