9
$\begingroup$

I have a single flat directory with over a million files. I just wanted to take a sample of the first few files but FileNames doesn't include a "only the first n" option, and so it took over a minute:

enter image description here

Is there a faster way?

$\endgroup$
8
$\begingroup$

Using OS shells commands seems to be much faster, although their output will need some massaging to obtain only the file names.

For instance, the following works quite well on my system (Win7-64):

Import["!dir /b /a-d C:\\Windows\\*", "Text"] // StringSplit[#, "\n"] &

This command takes 0.11s to execute; for comparison the corresponding FileNames expression (FileNames["*", "C:\\Windows\\"]) takes 0.33s. The difference is even more pronounced with larger/deeper directories.

In the dir command above, /a-d selects files that are not directories; /b produces minimal output which is easier to parse. If you want to traverse subdirectories, you can use the /s option.

$\endgroup$
  • $\begingroup$ I have OSX 10.11 $\endgroup$ – M.R. Jun 14 '16 at 18:37
  • $\begingroup$ @M.R. OK. So have you tried replacing the corresponding Unix-style shell commands in the Import expression? Does that not work? $\endgroup$ – MarcoB Jun 14 '16 at 18:44
  • $\begingroup$ @M. R., then you might want to look up the man page for ls. $\endgroup$ – J. M.'s technical difficulties Jun 14 '16 at 23:54
  • $\begingroup$ i think youll need find to list files-not-directories. ls will do fine for a specific extension though $\endgroup$ – george2079 Jun 15 '16 at 7:05
  • $\begingroup$ @george, or use the -p switch in ls. $\endgroup$ – J. M.'s technical difficulties Jun 15 '16 at 10:46
8
+25
$\begingroup$

New function in Mathematica 11 FileSystemMap with option MaxItems (documentation) can be useful here.

dir = "C:\\Users\\Alexey\\Documents";
n = 10;
f = FileSystemMap[#&, dir, MaxItems -> n] // Keys;
$\endgroup$
  • $\begingroup$ Try this on a large directory with, it hangs when you specify a recursive depth. n = 10; f = FileSystemMap[#&, dir, Infinity, MaxItems -> n] // Keys; $\endgroup$ – M.R. Sep 7 '16 at 17:55
  • $\begingroup$ @M.R. You are correct. This works not very good on a large directory. FileSystemMap and FileSystemScan are much slower than FileNames. It's very strange. $\endgroup$ – Alexey Golyshev Sep 10 '16 at 3:15
3
$\begingroup$

a linux version using find to list files and head to take the first n.

dir = "/path"
StringSplit[
    RunProcess[
         {"/bin/sh",
          "-c",
          "find "<>dir<>" -maxdepth 1 -type f | head -10"
          },"StandardOutput"]]

note find does not return files in any canonical order, you can sort for example, add | sort in front of | head

Note FileNames[][;;10]] is a good bit faster by my testing, but I don't have a folder with a million files to test it on.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.