# Import fails silently [closed]

In brief, an Import of a largish (11G) tsv file fails silently. I'm looking for ways to troubleshoot the problem further, and/or work around the failure.

Details

The machine I'm using has more than enough memory to handle such a large file. Below is a summary of the available memory before I start Mathematica:

$free -h total used free shared buffers cached Mem: 31G 1.5G 29G 6.1M 47M 298M -/+ buffers/cache: 1.2G 30G Swap: 15G 2.9G 12G  I start a fresh Mathematica session, and type in the expression shown below: (Note that at this point the word matrix is colored blue, indicating that the matrix is as-yet undefined.) I then press Shift+Enter, upon which • the cell's bracket changes from thin blue to thick black, and • the prefix "Running... " appears before the window's title in the notebook's title bar, ...and remain this way for several minutes. Eventually the cell's bracket's appearance reverts to its initial state, and the "Running... " prefix disappears from the window's title bar, but the notebook's appearance remains otherwise unchanged. In particular, • the color of the word matrix remains as it was before the evaluation; • no messages of any kind are visible. There are no messages in the Messages window (available through the menu command Windows > Messages), and no dialog window ever pops up. IOW, the evaluation appears to have failed, but there's no explicit mention of this fact. At this point, MaxMemoryUsed[] reports ~33MB (32612736 bytes, to be precise), a minuscule amount. (The kernel probably crashed during evaluation, so this figure is not very informative.) Indeed, the machine's memory usage now is not very different from what it was before: $ free -h
total       used       free     shared    buffers     cached
Mem:           31G       1.6G        29G       5.6M        24M       264M
-/+ buffers/cache:       1.3G        30G
Swap:          15G       2.9G        12G


FWIW, other programs I use regularly (e.g. MATLAB, ipython, python, perl, R, etc.) can handle this file without much problem.

Q: How can I troubleshoot the problem further?

Q: How can I get Mathematica to import this file?

• How much memory do you have in the computer? 32GB? I am a bit skeptical if M would be able to read such a huge tsv file and fit the result into memory. It won't even be a packed array. And even if it could read the file, it would probably take ages ... Of course I would love to be proven wrong about this. – Szabolcs Apr 28 '17 at 12:14
• @Szabolcs: yes, 32GB is what I have here. – kjo Apr 28 '17 at 12:15
• They say a better importer is coming in 11.2 community.wolfram.com/groups/-/m/t/1059289 – Szabolcs Apr 28 '17 at 12:29
• Do you need all the data contained on the file at the same time? I suggest to OpenRead and go line by line (ReadLine) parsing only the relevant information as needed in a way that can give you a hint where the readout fails. If its is because of memory, the reading order should not matter, if the file is corrupted, the location should be consistent. – rhermans Apr 28 '17 at 13:00
• whats in the file? If all numbers for example it may be better to use ReadList anyway – george2079 Apr 28 '17 at 15:35