2
$\begingroup$

In brief, an Import of a largish (11G) tsv file fails silently. I'm looking for ways to troubleshoot the problem further, and/or work around the failure.


Details

The machine I'm using has more than enough memory to handle such a large file. Below is a summary of the available memory before I start Mathematica:

$ free -h
             total       used       free     shared    buffers     cached
Mem:           31G       1.5G        29G       6.1M        47M       298M
-/+ buffers/cache:       1.2G        30G
Swap:          15G       2.9G        12G

I start a fresh Mathematica session, and type in the expression shown below:

Mathematica graphics

(Note that at this point the word matrix is colored blue, indicating that the matrix is as-yet undefined.)

I then press Shift+Enter, upon which

  • the cell's bracket changes from thin blue to thick black, and

  • the prefix "Running... " appears before the window's title in the notebook's title bar,

...and remain this way for several minutes.

Eventually the cell's bracket's appearance reverts to its initial state, and the "Running... " prefix disappears from the window's title bar, but the notebook's appearance remains otherwise unchanged.

In particular,

  • the color of the word matrix remains as it was before the evaluation;

  • no messages of any kind are visible.

There are no messages in the Messages window (available through the menu command Windows > Messages), and no dialog window ever pops up.

IOW, the evaluation appears to have failed, but there's no explicit mention of this fact.

At this point, MaxMemoryUsed[] reports ~33MB (32612736 bytes, to be precise), a minuscule amount. (The kernel probably crashed during evaluation, so this figure is not very informative.) Indeed, the machine's memory usage now is not very different from what it was before:

$ free -h
             total       used       free     shared    buffers     cached
Mem:           31G       1.6G        29G       5.6M        24M       264M
-/+ buffers/cache:       1.3G        30G
Swap:          15G       2.9G        12G

FWIW, other programs I use regularly (e.g. MATLAB, ipython, python, perl, R, etc.) can handle this file without much problem.

Q: How can I troubleshoot the problem further?

Q: How can I get Mathematica to import this file?

$\endgroup$
  • 2
    $\begingroup$ How much memory do you have in the computer? 32GB? I am a bit skeptical if M would be able to read such a huge tsv file and fit the result into memory. It won't even be a packed array. And even if it could read the file, it would probably take ages ... Of course I would love to be proven wrong about this. $\endgroup$ – Szabolcs Apr 28 '17 at 12:14
  • $\begingroup$ @Szabolcs: yes, 32GB is what I have here. $\endgroup$ – kjo Apr 28 '17 at 12:15
  • 1
    $\begingroup$ They say a better importer is coming in 11.2 community.wolfram.com/groups/-/m/t/1059289 $\endgroup$ – Szabolcs Apr 28 '17 at 12:29
  • 1
    $\begingroup$ Do you need all the data contained on the file at the same time? I suggest to OpenRead and go line by line (ReadLine) parsing only the relevant information as needed in a way that can give you a hint where the readout fails. If its is because of memory, the reading order should not matter, if the file is corrupted, the location should be consistent. $\endgroup$ – rhermans Apr 28 '17 at 13:00
  • 1
    $\begingroup$ whats in the file? If all numbers for example it may be better to use ReadList anyway $\endgroup$ – george2079 Apr 28 '17 at 15:35