2 deleted 3 characters in body
source | link

OK, I found a simple solution, at least if grep or the like is available. Here are the timing results when I apply to the file I used in my original post:

stringsrows = 
  ReadList["!grep -iv nan " <> pathToTSV, types, 
    RecordSeparators -> {tab, nl}] // printTime;

3.72033 s   ReadList[!grep -v nan <> pathToTSV, types, RecordSeparators -> {tab, nl}]

One could generalize this idea to any type of pre-processing pipe (e.g. to weed out any line that contains anything other than numbers, not just NaNs), though, of course, all pre-processing incurs a performance cost.

OK, I found a simple solution, at least if grep or the like is available. Here are the timing results when I apply to the file I used in my original post:

strings = 
  ReadList["!grep -iv nan " <> pathToTSV, types, 
    RecordSeparators -> {tab, nl}] // printTime;

3.72033 s   ReadList[!grep -v nan <> pathToTSV, types, RecordSeparators -> {tab, nl}]

One could generalize this idea to any type of pre-processing pipe (e.g. to weed out any line that contains anything other than numbers, not just NaNs), though, of course, all pre-processing incurs a performance cost.

OK, I found a simple solution, at least if grep or the like is available. Here are the timing results when I apply to the file I used in my original post:

rows = 
  ReadList["!grep -iv nan " <> pathToTSV, types, 
    RecordSeparators -> {tab, nl}] // printTime;

3.72033 s   ReadList[!grep -v nan <> pathToTSV, types, RecordSeparators -> {tab, nl}]

One could generalize this idea to any type of pre-processing pipe (e.g. to weed out any line that contains anything other than numbers, not just NaNs), though, of course, all pre-processing incurs a performance cost.

1
source | link

OK, I found a simple solution, at least if grep or the like is available. Here are the timing results when I apply to the file I used in my original post:

strings = 
  ReadList["!grep -iv nan " <> pathToTSV, types, 
    RecordSeparators -> {tab, nl}] // printTime;

3.72033 s   ReadList[!grep -v nan <> pathToTSV, types, RecordSeparators -> {tab, nl}]

One could generalize this idea to any type of pre-processing pipe (e.g. to weed out any line that contains anything other than numbers, not just NaNs), though, of course, all pre-processing incurs a performance cost.