5
$\begingroup$

I have a directory on my portable hard drive, with around 350000 files under it (there is no subdirectory, only files). I want to get all the file dirs so use

 filedirs=FileNames["*",mydir]

However, the code above run for more than 1 hour and still not return a result. Why is FileNames so slow? What should I do?

I'm using Mathematica 12.0 on my Win10.

The problem is bypassed by running tree dir /f /a >outputtxt on win10's cmd window, as suggested by @mikado. The size of outputtxt is only ~20M and it takes only several minites to get result. I still don't understand why FileNames will take astonishing time to run.

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ Two ideas come to mind: 1) the OS is just overloaded by having so many files in a single flat structure 2) Mathematica's struggling to keep 350000 file name strings in memory. Both are plausible and both have a simple solution: use folders. $\endgroup$
    – b3m2a1
    Commented Jul 23, 2020 at 4:17
  • $\begingroup$ emmm........how to use folders? I have to manully copy and move thousands of files?@b3m2a1 $\endgroup$
    – Harry
    Commented Jul 23, 2020 at 4:21
  • $\begingroup$ Use an external tool to write the names to file, then get Mathematica to read the file $\endgroup$
    – mikado
    Commented Jul 23, 2020 at 5:51
  • $\begingroup$ good idea! In windows cmd run tree dir /f /a >outputtxt I get the filename list. @mikado $\endgroup$
    – Harry
    Commented Jul 23, 2020 at 6:41
  • $\begingroup$ @b3m2a1 In my experience, it's not Mathathematica that struggles from having to handle these strings. I've seen this problem as well and I think it's the OS or the interface between the two. $\endgroup$
    – Sjoerd Smit
    Commented Jul 23, 2020 at 7:08

1 Answer 1

Reset to default
3
$\begingroup$

Under windows system this is the way to get huge number of filenames under a directory:

filenamesFast[dir_]:=
Module[{txtname="\\temp-alldirs.txt"},
Run["dir \""<>dir<>"\" /b >\""<>dir<>txtname<>"\""];
(dir<>#)&/@StringSplit[Import[dir<>txtname],"\n"]
]
Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ This is very useful. If you use Import[..., "List"], you don't need to do the StringSplit step. I also recommend using FileNameJoin instead of joining file paths with StringJoin. $\endgroup$
    – Sjoerd Smit
    Commented Sep 24, 2020 at 8:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.