0
$\begingroup$

I have a notebook which takes 24 hours or so for evaluation. The notebook uses Export command to output a dat file with two columns of data.

Is there any way to access the data that is currently being produced? I mean to say that while the notebook has not completed evaluation I need the data generated, perhaps it is in some temporary file hidden by Mathematica and then it will be written to the dat file later on when evaluation is complete. I have looked into the Mathematica directory, hidden files, temporary files but couldn't identify something that would resemble what I require.

I am using Mathematica 11.0 on Linux Mint. Any help or comments are greatly appreciated.

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ notebook uses Export command first of all, you should always specify the full path of your export file, so you know where it is. Or use SetDirectory[..] at the start. If you do not, then it should go to whatever Directory[] shows. On windows for example, it points to "C:\\Users\\userName\\Documents" If you do SetDirectory[NotebookDirectory[]] then this points to same folder where the notebook is. $\endgroup$ – Nasser Apr 22 '20 at 7:48
  • $\begingroup$ @Nasser Sorry, my question was a little vague. I have updated it to reflect my thoughts better. Please check it. $\endgroup$ – Nitin Apr 22 '20 at 8:04
  • $\begingroup$ There are no hidden files or anything. The data goes to the file itself in the export. It might not have flushed out to the file when you are looking at it? On windows, just opening the file being written to using external text editor forces the data to show for example. Have you tried this on linux? May be you could make a MWE showing your small LOOP and example how you doing the export to the file. $\endgroup$ – Nasser Apr 22 '20 at 8:15
  • $\begingroup$ You could also have a look at Monitor. $\endgroup$ – Fred Simons Apr 22 '20 at 9:07
0
$\begingroup$

The file is written when Export is evaluated. I assume your code does not spend all that time evaluating Export, but in generating the data. Thus, no, the data is not on disk, and you won't find it there.

You probably want to output the data gradually as it is being computed instead of in one step at the end. Then you may be able to look at the file and see how data is being appended to it.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Yes, that's exactly what I want. Before I accept the answer can you please also tell me if it doing this will slow down the overall evaluation time? On a scale of change from 12 hours to 13 hours I would be perfectly fine with this but if this adds more than 2 3 hours to the overall evaluation time then it won't be beneficial to me. Another thing is, the Mathematica must be storing the data somewhere before writing it to the file at the very end of 20 hours (or whatever), right? Some kind of buffer or something atleast? $\endgroup$ – Nitin Apr 22 '20 at 8:53
  • $\begingroup$ It's stored in memory. $\endgroup$ – Szabolcs Apr 22 '20 at 9:06
  • $\begingroup$ There is an overhead to calling Export. Whether it will slow down the calculation depends on how many times you call it, and whether Export is slower than the calculations done between two Export calls. That seems very unlikely if the whole thing takes 12 hours. I wouldn't expect it to slow down, but it's impossible to be sure without knowing what you are doing exactly. $\endgroup$ – Szabolcs Apr 22 '20 at 9:08
  • $\begingroup$ Isn't there any way to access the stored data in memory? $\endgroup$ – Nitin Apr 22 '20 at 9:41
  • 2
    $\begingroup$ I don't think we can go deeper into this without a representative small example of what you are doing (sscce.org) $\endgroup$ – Szabolcs Apr 22 '20 at 9:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.