2
$\begingroup$

In brief, an Import of a largish (11G) tsv file fails silently. I'm looking for ways to troubleshoot the problem further, and/or work around the failure.


Details

The machine I'm using has more than enough memory to handle such a large file. Below is a summary of the available memory before I start Mathematica:

$ free -h
             total       used       free     shared    buffers     cached
Mem:           31G       1.5G        29G       6.1M        47M       298M
-/+ buffers/cache:       1.2G        30G
Swap:          15G       2.9G        12G

I start a fresh Mathematica session, and type in the expression shown below:

Mathematica graphics

(Note that at this point the word matrix is colored blue, indicating that the matrix is as-yet undefined.)

I then press Shift+Enter, upon which

  • the cell's bracket changes from thin blue to thick black, and

  • the prefix "Running... " appears before the window's title in the notebook's title bar,

...and remain this way for several minutes.

Eventually the cell's bracket's appearance reverts to its initial state, and the "Running... " prefix disappears from the window's title bar, but the notebook's appearance remains otherwise unchanged.

In particular,

  • the color of the word matrix remains as it was before the evaluation;

  • no messages of any kind are visible.

There are no messages in the Messages window (available through the menu command Windows > Messages), and no dialog window ever pops up.

IOW, the evaluation appears to have failed, but there's no explicit mention of this fact.

At this point, MaxMemoryUsed[] reports ~33MB (32612736 bytes, to be precise), a minuscule amount. (The kernel probably crashed during evaluation, so this figure is not very informative.) Indeed, the machine's memory usage now is not very different from what it was before:

$ free -h
             total       used       free     shared    buffers     cached
Mem:           31G       1.6G        29G       5.6M        24M       264M
-/+ buffers/cache:       1.3G        30G
Swap:          15G       2.9G        12G

FWIW, other programs I use regularly (e.g. MATLAB, ipython, python, perl, R, etc.) can handle this file without much problem.

Q: How can I troubleshoot the problem further?

Q: How can I get Mathematica to import this file?

$\endgroup$

closed as off-topic by MarcoB, Henrik Schumacher, Coolwater, m_goldberg, Sektor May 30 '18 at 8:42

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question cannot be answered without additional information. Questions on problems in code must describe the specific problem and include valid code to reproduce it. Any data used for programming examples should be embedded in the question or code to generate the (fake) data must be included." – MarcoB, Henrik Schumacher, Coolwater, m_goldberg, Sektor
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    $\begingroup$ How much memory do you have in the computer? 32GB? I am a bit skeptical if M would be able to read such a huge tsv file and fit the result into memory. It won't even be a packed array. And even if it could read the file, it would probably take ages ... Of course I would love to be proven wrong about this. $\endgroup$ – Szabolcs Apr 28 '17 at 12:14
  • $\begingroup$ @Szabolcs: yes, 32GB is what I have here. $\endgroup$ – kjo Apr 28 '17 at 12:15
  • 1
    $\begingroup$ They say a better importer is coming in 11.2 community.wolfram.com/groups/-/m/t/1059289 $\endgroup$ – Szabolcs Apr 28 '17 at 12:29
  • 1
    $\begingroup$ Do you need all the data contained on the file at the same time? I suggest to OpenRead and go line by line (ReadLine) parsing only the relevant information as needed in a way that can give you a hint where the readout fails. If its is because of memory, the reading order should not matter, if the file is corrupted, the location should be consistent. $\endgroup$ – rhermans Apr 28 '17 at 13:00
  • 1
    $\begingroup$ whats in the file? If all numbers for example it may be better to use ReadList anyway $\endgroup$ – george2079 Apr 28 '17 at 15:35