How do I import a 1 gig text file by only importing the last 3 lines of every 10 line chunk?

Question

I am trying to get the last three lines of every 10 line chunk into mathematica as a data variable. The text file has 3 columns. For example:

1 1 1
2 2 2
3 3 3  
4 4 4 
5 5 5 
6 6 6 
7 7 7
8 8 8 
9 9 9
10 10 10 
11 11 11 
12 12 12 
13 and so on
14
15
16
17
18
19
20

I need in Mathematica to be the following:

 {{8, 8, 8}, {9, 9, 9}, {10, 10, 10}, {18, 18, 18}, {19, 19, 19}, {20, 20, 20}}

So far, I have tried using data = ReadList["file.txt",{Real,Real,Real}] to manually import them all and select them after that, but this file is very large and that process takes too much memory for my machine. Is there any way using streams to import these specific lines? Additionally, can this file be read and directly put into a "file.mx" so that the data takes a lot less memory. I have tried doing DumpSave["file.mx",data] once I had the specific data I wanted, but it doubled the memory usage of Mathematica.

Answer 1

Maybe something along these lines would work?

columns = 3;
skiprows = 7;
readrows = 3;
stream = OpenRead["~/Desktop/Example.txt"];
data = Flatten[
   Reap[
     While[
      True,
      If[Skip[stream, Real, columns*skiprows] == EndOfFile, Break[]];
      lines = Partition[ReadList[stream, Real, columns*readrows], columns];
      If[
       Length[lines] > 0,
       Sow[lines],
       Break[]
       ]
      ]
     ][[2, 1]],
   1];
Close[stream];

Basically, I'm skipping the 21 numbers that you don't want with Skip, reading in the 9 that you do with ReadList, and then using Sow and Reap to collect them. For large collections of initially unknown size, Sow and Reap are much faster than Append or AppendTo. If you know the exact number of elements, you might be able to eke a bit more speed out of it by initially creating an array of the appropriate size.

I'm checking during both the Skip and just after ReadList to see if I've reached the end of the file. If you know beforehand how many lines are in your file, you could switch to a Do loop and run it $n$ times.

Hopefully I'm not missing any important edge cases.

Answer 2

data.txt

1 1 1
2 2 2
3 3 3  
4 4 4 
5 5 5 
6 6 6 
7 7 7
8 8 8 
9 9 9
10 10 10 
11 11 11 
12 12 12
13 13 13
14 14 14
15 15 15
16 16 16
17 17 17
18 18 18
19 19 19
20 20 20
21 21 21
22 22 22
23 23 23
24 24 24
25 25 25
26 26 26
27 27 27
28 28 28
29 29 29

Method 1

data = Import["data.txt", "Table"]
Drop[#, UpTo[7]] & /@ Partition[data, UpTo[10]] // Flatten[#, 1] &

Method 2

data = Import["data.txt", "Table"]
Part[data, 
 Select[Range[Length@data], 
  Mod[#, 10, 1] == 8 || Mod[#, 10, 1] == 9 || Mod[#, 10, 1] == 10 &]]

{{1, 1, 1}, {2, 2, 2}, {3, 3, 3}, {4, 4, 4}, {5, 5, 5}, {6, 6, 6}, {7,7, 7}, {8, 8, 8}, {9, 9, 9}, {10, 10, 10}, {11, 11, 11}, {12, 12, 12}, {13, 13, 13}, {14, 14, 14}, {15, 15, 15}, {16, 16, 16}, {17, 17, 17}, {18, 18, 18}, {19, 19, 19}, {20, 20, 20}, {21, 21, 21}, {22, 22, 22}, {23, 23, 23}, {24, 24, 24}, {25, 25, 25}, {26, 26, 26}, {27, 27, 27}, {28, 28, 28}, {29, 29, 29}}

{{8, 8, 8}, {9, 9, 9}, {10, 10, 10}, {18, 18, 18}, {19, 19, 19}, {20,20, 20}, {28, 28, 28}, {29, 29, 29}}