Skipping commented in a stream from OpenRead

Question

Is there a solution to skip comments in text files that are imported with OpenRead? I know there are solutions for Importand ImportStringbut I cannot find any way to handle the streams. Imagine I have the file:

#comment
#comment
  1 2 3
 # another comment
  4 5 6
  7 8 9
  1 3 4
# another comment

Also notice that there are whitespaces before comments and before data. Finally, the comments are effectively at random places such that no repetition pattern can be inferred a priori.

I was thinking of Skipbut it seems to have only fixed types so I probably cannot use a pattern matching. There is a similar solution here but it does not address the specific situation when a line starts with a specific character. At this moment I am thinking of either implementing the slow Import or preprocessing with Python but I bet there must be a simple way with Mathematica.

A second problem whether I can use the stream to extract data right away (in the example above I would like to know the values in columns 2 and 3 if the value in the first column equals 1. I know how to do this with data from Importbut I am not sure about how to do it with the stream. I found some code utilizing While and EndOfFile but as everybody says do not use loops in Mathematica, what else is available?

Also, it would be great if the whole solution was parallelizable (I need to read large number of files, process them --- extract data --- then put the extracted data into a common list).

Answer 1

OpenRead itself does not read files. It just opens a stream. Then you can use other functions such as Import, Read, ReadList, etc. to read from that stream.

As I understand, what you want achieve is to filter the stream while still keeping it as a stream (and not, say, a list of strings representing lines). I think this is possible with stream methods, but I have never done it, and it doesn't look quick or easy to implement it in a performance way.

So instead I recommend to process the file through an external program such as sed, and pipe the data to Mathematica. This works on Windows too, but you need to install sed and add its location to the operating system's path.

Examples

Your file:

Import["~/Desktop/file.txt", "String"]

"#comment
#comment
  1 2 3
 # another comment
  4 5 6
  7 8 9
  1 3 4
# another comment
"

Filter it with sed:

Import["!sed /^[[:space:]]*#/d ~/Desktop/file.txt", "String"]

"  1 2 3
  4 5 6
  7 8 9
  1 3 4
"

Filter with sed and use as stream:

stream = OpenRead["!sed /^[[:space:]]*#/d ~/Desktop/file.txt"]

Import[stream, "String"]

"  1 2 3
  4 5 6
  7 8 9
  1 3 4
"

Close[stream]