11
$\begingroup$

I've got code that's exporting a file, but the file might be so big that it will take up too much space on the cluster I'm using and crash other users' processes. Is there anything like MemoryConstrained, but for disk space? Or any way in general to restrict the disk space Export is allowed to use? I'd like the file to export if it would take less than a certain amount of disk space, but not export if it would take more. Is there any way to do this?

$\endgroup$
  • $\begingroup$ A possible very rough workaround is TimeConstrained or ByteCount because both are correlated with the output size. $\endgroup$ – ybeltukov Oct 19 '14 at 15:04
  • $\begingroup$ The only reliable way I can suggest is to write your own exporter, which would export in chunks. In such a case, you will be able to inject such constraint into the logic of your exporter. It might be not as hard as it may sound, since in many cases you can use ExportString to export individual chunks (e.g. this is the case for CSV). $\endgroup$ – Leonid Shifrin Oct 19 '14 at 16:18
  • $\begingroup$ If[StringLength[ExportString[...,...]] > n, ..., ...] wouldn't work? Or would that use up too much memory (ie, it'd use the same amount of memory as it would diskspace) $\endgroup$ – barrycarter Oct 19 '14 at 17:04
  • 2
    $\begingroup$ @Matt, I think the answer may depend on the details, what format is the file you want to export? $\endgroup$ – rhermans Oct 19 '14 at 18:32
  • $\begingroup$ @rhermans It's a .m file. barrycarter's solution sounds like it would work, since MemoryConstrained could keep it from crashing the cluster. I'll try it and see if it works. $\endgroup$ – Matt Oct 20 '14 at 15:25
1
$\begingroup$

I would suggest the workaround to create smaller files and use the sum of

FileByteCount[]

of each file as a check while exporting.

$\endgroup$
  • 1
    $\begingroup$ Wouldn't you want to know the size of the file before attempting to write it? I'm thinking one would want to know the amount of free space on the filesystem. $\endgroup$ – Michael E2 Aug 11 '15 at 17:38
  • $\begingroup$ checking the space on the filesys is also possible: mathematica.stackexchange.com/questions/16804/… StringJoin[ReadList["!dir c:\ ", Word, RecordLists -> True,WordSeparators -> {" ", ","}][[-1, 3 ;; -3]]] // ToExpression $\endgroup$ – MatrixBreaker Aug 12 '15 at 6:42
  • $\begingroup$ If you also need to have an approximate filesize, it should be good to call ByteCount on whatever you are exporting, as ybeltukov suggested $\endgroup$ – MatrixBreaker Aug 12 '15 at 6:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.