Getting line number based on regexp match or Importing data with variable header size

Question

I have a text file with structure:

line 1 with some words
another line with some \\\ structure or ///
etc
until the last word line
lastline and now numbers which can be separated by anything
1232.232 12312.32
123.22
...

I would like to figure out where the data starts. The most obvious thing (but probably not the best one) would be to match the regexp '^lastline.+$' and return its line number. But this appears to be quite complicated, I have not found any method to get line numbers easily.

I am using the following to import the data:

Import[mytextfile, "Table", HeaderLines -> line_number_of_last_line]

There is a post here but it seems to assume that all the lines to skip have the comment sign # while I only know how the last line to skip looks like (the last line before data). So I would actually like a solution based on the logic read everything from the line folowing the regexp match.

Answer 1

So I would actually like a solution based on the logic read everything from the line folowing the regexp match.

You can skip the header and then import the rest of your file:

importData[stream_] := Module[{data},
   While[With[{rec = Read[stream, String]}, 
     rec =!= EndOfFile && ! StringStartsQ[rec, "lastline"]]];
   data = Import[stream, "Table"];
   Close[stream];
   data];

Now

txt = "line 1 with some words
another line with some \\\\\\ structure or ///
etc
until the last word line
lastline and now numbers which can be separated by anything
1232.232 12312.32
123.22";
stream = StringToStream[txt];

importData[stream]

{{1232.23, 12312.3}, {123.22}}

Answer 2

You may use a procedural approach with streams to locate the last line since you know its format. This should be quicker than the regex approach for large files.

With

txt =
"line 1 with some words
another line with some \\\\\\ structure or ///
etc
until the last word line
lastline and now numbers which can be separated by anything
1232.232 12312.32
123.22";

Then the below will locate the "lastline" line number. I use StringToStream but you would use OpenRead with the file path of the file. StringStartsQ can be swapped out for any test that complements the format of the last line.

header =
 Module[{stream, lastLine = 1},
  stream = StringToStream[txt];
  While[
   With[{rec = Read[stream, Record]},
    rec =!= EndOfFile && ! StringStartsQ[rec, "lastline"]
    ],
   lastLine++];
  Close[stream];
  lastLine
  ]

5

and

ImportString[txt, "Table", HeaderLines -> header]

{{1232.23, 12312.3}, {123.22}}

Hope this helps.