4
$\begingroup$

Currently we have a (not very efficient, but still) function FileNames allowing to get the complete listing of all files and subdirectories of a directory at arbitrary depth. But it has no option allowing to restrict its output to the actual files when directories should be excluded. Hence for selecting only files from its output we are forced to use DirectoryQ, FileType or undocumented FileInformation, all of them don't accept a list as an argument and add significant time to FileNames:

FileNames[All, $InstallationDirectory, Infinity]; // AbsoluteTiming
FileNames[All, $InstallationDirectory, Infinity]; // AbsoluteTiming
l1 = Select[FileNames[All, $InstallationDirectory, Infinity], 
    Not@*DirectoryQ]; // AbsoluteTiming
l2 = Select[FileNames[All, $InstallationDirectory, Infinity], 
    FileType[#] === File &]; // AbsoluteTiming
l3 = Select[FileNames[All, $InstallationDirectory, Infinity], 
    FileInformation[#, "FileType"] === "File" &]; // AbsoluteTiming
l4 = Select[FileNames[All, $InstallationDirectory, Infinity], 
    Information[File@#, "FileType"] === "File" &]; // AbsoluteTiming

l1 == l2 == l3 == l4
{7.1199, Null}

{6.98109, Null}

{12.113, Null}

{22.7185, Null}

{20.8688, Null}

{397.981, Null}

True

Is there more efficient way to obtain the complete listing of only actual files in a directory and its subdirectories at arbitrary depth?

$\endgroup$
  • 1
    $\begingroup$ There is also Keys@FileSystemMap[# &, $InstallationDirectory, Infinity, 1] but it is slower for me than to use FileNames. $\endgroup$ – C. E. Aug 25 '19 at 8:03
2
$\begingroup$

One way is to use the system dir command. The disadvantage is that it doesn't allow to specify the arbitrary depth (at least on Windows). For Windows:

FileNames[All, $InstallationDirectory, Infinity]; // AbsoluteTiming
l1 = Select[FileNames[All, $InstallationDirectory, Infinity], 
    Not@*DirectoryQ]; // AbsoluteTiming
l5 = ImportString[
    RunProcess[{"cmd", "/c", "dir", "/b", "/s", "/a-d", $InstallationDirectory}, 
     "StandardOutput"], "Lines", 
    CharacterEncoding -> $SystemCharacterEncoding]; // AbsoluteTiming

l1 == Sort@l5    
{7.16807, Null}

{14.5409, Null}

{3.55901, Null}

True

Another possible way which allows to specify arbitrary depth is to use Windows PowerShell as shown in this thread. This answer shows how it can be used from Mathematica.


I've tried to reduce the filesystem load by filtering out directory names (found by applying DirectoryName) from the list returned by FileNames. It didn't gave sensible speedup however because in the case of $InstallationDirectory this list contains only about 10% of directories. But when working with network paths it can give appreciable speedup, so I publish here this solution too:

l6 = Module[{l = FileNames[All, $InstallationDirectory, Infinity], NonEmptyDirectories, 
     lFiltered},
    (* StringDrop is needed only under Windows! It drops the ending "\\". *)
    NonEmptyDirectories = StringDrop[DeleteDuplicates[DirectoryName /@ l], -1];
    lFiltered = DeleteCases[l, Alternatives @@ NonEmptyDirectories];
    Select[lFiltered, Not@*DirectoryQ]]; // AbsoluteTiming

l1 == l6
{12.4957, Null}

True

It should be noted that there are 21 empty and 4181 non-empty directories in $InstallationDirectory and 44681 actual files. We are forced to check 44702 filepaths just to remove these 21 empty directories from the list...

$\endgroup$
  • $\begingroup$ Similar function for Unix: listFiles[root_] := ImportString[RunProcess[{"find", root, "-type", "f"}, "StandardOutput"], "Lines"] It finds 42718 files though, whereas the other methods find 42986. By the way, how many files do you have in your $InstallationDirectory? I ask because for me, e.g. l1 and l2 can be computed in ~1 second. l4 in 12 seconds. The bash version executes in 0.58 seconds. $\endgroup$ – C. E. Aug 25 '19 at 18:26
  • $\begingroup$ @C.E. I get 44681 filenames with version 12.0 on Windows 7 x64. $\endgroup$ – Alexey Popkov Aug 25 '19 at 18:33
  • $\begingroup$ @C.E. It seems Unix has its own dir utility which has functional similar to cmd's dir, but more straightforward syntax. $\endgroup$ – Alexey Popkov Aug 25 '19 at 18:46
1
$\begingroup$

If you don't mind installing the free linux emulator Cygwin on your Windows 7 machine and make sure that bash.exe is in your path, then you have the option of running linux commands on all platforms. Here is an example of counting all the files and counting all the files to depth 2.

RunProcess[{"bash", "-c", "/usr/bin/find . -type f | wc -l"},
   ProcessDirectory -> $InstallationDirectory] // AbsoluteTiming
RunProcess[{"bash", "-c", 
   "/usr/bin/find . -maxdepth 2 -type f | wc -l"}, 
  ProcessDirectory -> $InstallationDirectory] // AbsoluteTiming
(* {2.13029, <|"ExitCode" -> 0, "StandardOutput" -> "44678
   ", "StandardError" -> ""|>} *)
(* {0.125235, <|"ExitCode" -> 0, "StandardOutput" -> "11
   ", "StandardError" -> ""|>} *)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.