4
$\begingroup$

I have no reason to suspect this will exist, but I hope it does.

Is there a nice way to efficiently count the cumulative file size of a directory in Mathematica?

I know I can do something like this:

Select[FileNames["*", dir, Infinity], 
   Not@*DirectoryQ] // Map[FileByteCount] // Total

But when the directories get big this gets slow:

Select[FileNames["*", "~/Documents", Infinity], Not@*DirectoryQ] // 
   Replace[
     Quiet@Map[FileByteCount, #],
     Except[_Integer] -> 0,
     1
     ] & // Total // AbsoluteTiming

{2.38783, 6161484250}

I know I could pass this to the OS:

ToExpression@
     StringSplit[
       RunProcess[{"du",  "-s", ExpandFileName@#}, 
        "StandardOutput"]][[1]]*1024/2 &@
  "~/Documents" // AbsoluteTiming

{0.352957, 6080770048}

But it'd be nice to have a platform independent way to do it similarly fast.

Is there such a way?

$\endgroup$

1 Answer 1

5
$\begingroup$

You can do this via JLink, but that would need a bit of preparation. First, we load JLink:

<<JLink`

Now download and add to the Java class path the apache commons IO library:

AddToClassPath[
  URLDownload @ "http://central.maven.org/maven2/commons-io/commons-io/2.5/commons-io-2.5.jar"
]

Here is the function to use then:

ClearAll[directoryByteCount];
directoryByteCount[dir_String?DirectoryQ]:=
  JavaBlock[
    LoadJavaClass["org.apache.commons.io.FileUtils"];
    FileUtils`sizeOfDirectory[
      JavaNew["java.io.File", AbsoluteFileName @ dir]
    ]
  ];

For example, on my machine I get (after a first couple of runs which take a bit longer since the class has to be loaded and JVM warms up):

directoryByteCount @ "~/Documents" // AbsoluteTiming

(* {0.326422, 55360848865}  *)

Since this is Java, you should get platform independence as requested (although I have only tested this on Mac OS so far).

$\endgroup$
4
  • $\begingroup$ I'm not getting the same speed-up over time. 12 calls in I still get timings of ~5 seconds. Any idea what that is? $\endgroup$
    – b3m2a1
    Nov 25, 2017 at 3:20
  • $\begingroup$ @b3m2a1 For me, it was 6 sec, 0.5 sec, 0.3 sec, and then consistently 0.3 secs. Your method with du is still faster though - about 0.05 seconds on my machine. But Java implementation is supposed to be optimized, and JLink overhead per call is only about 0.005 to 0.01 seconds. So no, no idea why this is so slow for you. $\endgroup$ Nov 25, 2017 at 16:03
  • $\begingroup$ It's likely something JVM configuration related, anyway. And most likely localized to my machine. The answer fits the bill for what I was looking for. $\endgroup$
    – b3m2a1
    Nov 25, 2017 at 21:18
  • $\begingroup$ @b3m2a1 Re: "It's likely something JVM configuration related, anyway. And most likely localized to my machine" May be. But such a dramatic speed difference as what you report vs. what I get is still rather strange. The operation should not be memory-hungry in java, so is unlikely to be affected by the java heap size setting. Anyway, I am glad if this helps, and thanks for the accept. $\endgroup$ Nov 25, 2017 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.