Stream reading 1000 times faster in Python

Question

I have a 26GB CSV. I want to extract all the rows that have data for a specific sensor. In Mathematica I write:

stream = OpenRead[my26GBFile];
Reap[Block[{line},
  line = ReadLine[stream];
  While[
    StringQ[line],
    If[StringContainsQ[line, "Sensor123"], Sow[line]]
    line = ReadLine[stream]
  ]
]]

I let it run overnight because it takes at least an hour. I wrote the following in Python and it ran in about 2 minutes. I would not expect the performance difference to be that great. Anyone have insight into what's happening and how to get better performance in Mathematica?

import csv

with open(my26GBFile, 'r') as csv_file:
  csv_reader = csv.reader(csv_file)

  with open(newCSVFile, 'w') as new_file:
    csv_writer = csv.writer(new_file, delimiter=',')
    for line in csv_reader:
      if 'Sensor123' in line:
        csv_writer.writerow(line)
      else:
        continue

  new_file.close()
csv_file.close()

Answer 1

FindList is meant to deal with such tasks. Not sure whether it works with all types of files but for plaintext files like in the case of CSV file it should work.

Answer 2

With python you have GIL, which makes a lot of things run slower.

You could use a library like polars to speed things up. e.g.

import polars as pl
def cell(filepath):
    df = pl.read_csv()
    return df.to_dicts() # turns dataframe into list of dict