11
$\begingroup$

I've got code that's exporting a file, but the file might be so big that it will take up too much space on the cluster I'm using and crash other users' processes. Is there anything like MemoryConstrained, but for disk space? Or any way in general to restrict the disk space Export is allowed to use? I'd like the file to export if it would take less than a certain amount of disk space, but not export if it would take more. Is there any way to do this?

Improve this question
$\endgroup$
5
  • $\begingroup$ A possible very rough workaround is TimeConstrained or ByteCount because both are correlated with the output size. $\endgroup$
    – ybeltukov
    Commented Oct 19, 2014 at 15:04
  • $\begingroup$ The only reliable way I can suggest is to write your own exporter, which would export in chunks. In such a case, you will be able to inject such constraint into the logic of your exporter. It might be not as hard as it may sound, since in many cases you can use ExportString to export individual chunks (e.g. this is the case for CSV). $\endgroup$
    – Leonid Shifrin
    Commented Oct 19, 2014 at 16:18
  • $\begingroup$ If[StringLength[ExportString[...,...]] > n, ..., ...] wouldn't work? Or would that use up too much memory (ie, it'd use the same amount of memory as it would diskspace) $\endgroup$
    – user1722
    Commented Oct 19, 2014 at 17:04
  • 2
    $\begingroup$ @Matt, I think the answer may depend on the details, what format is the file you want to export? $\endgroup$
    – rhermans
    Commented Oct 19, 2014 at 18:32
  • $\begingroup$ @rhermans It's a .m file. barrycarter's solution sounds like it would work, since MemoryConstrained could keep it from crashing the cluster. I'll try it and see if it works. $\endgroup$
    – Matt
    Commented Oct 20, 2014 at 15:25

1 Answer 1

Reset to default
1
$\begingroup$

I would suggest the workaround to create smaller files and use the sum of

FileByteCount[]

of each file as a check while exporting.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Wouldn't you want to know the size of the file before attempting to write it? I'm thinking one would want to know the amount of free space on the filesystem. $\endgroup$
    – Michael E2
    Commented Aug 11, 2015 at 17:38
  • $\begingroup$ checking the space on the filesys is also possible: mathematica.stackexchange.com/questions/16804/… StringJoin[ReadList["!dir c:\ ", Word, RecordLists -> True,WordSeparators -> {" ", ","}][[-1, 3 ;; -3]]] // ToExpression $\endgroup$
    – MatrixBreaker
    Commented Aug 12, 2015 at 6:42
  • $\begingroup$ If you also need to have an approximate filesize, it should be good to call ByteCount on whatever you are exporting, as ybeltukov suggested $\endgroup$
    – MatrixBreaker
    Commented Aug 12, 2015 at 6:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.