15
$\begingroup$

I have a 26GB CSV. I want to extract all the rows that have data for a specific sensor. In Mathematica I write:

stream = OpenRead[my26GBFile];
Reap[Block[{line},
  line = ReadLine[stream];
  While[
    StringQ[line],
    If[StringContainsQ[line, "Sensor123"], Sow[line]]
    line = ReadLine[stream]
  ]
]]

I let it run overnight because it takes at least an hour. I wrote the following in Python and it ran in about 2 minutes. I would not expect the performance difference to be that great. Anyone have insight into what's happening and how to get better performance in Mathematica?

import csv

with open(my26GBFile, 'r') as csv_file:
  csv_reader = csv.reader(csv_file)

  with open(newCSVFile, 'w') as new_file:
    csv_writer = csv.writer(new_file, delimiter=',')
    for line in csv_reader:
      if 'Sensor123' in line:
        csv_writer.writerow(line)
      else:
        continue

  new_file.close()
csv_file.close()
$\endgroup$
8
  • 3
    $\begingroup$ I'm curious, can you time grep Sensor123 my26GBFile > newCSVFile and does the result exactly match the output of MMA and Python? Thanks $\endgroup$
    – Bill
    Jan 18 at 20:39
  • 1
    $\begingroup$ @Bill Using grep in terminal: 2:21. Using Python: 3:08. Using Mathematica: Maybe 100 minutes $\endgroup$ Jan 18 at 20:46
  • 4
    $\begingroup$ Have you tried FindList? reference.wolfram.com/legacy/language/v13/ref/FindList.html $\endgroup$ Jan 18 at 21:03
  • 2
    $\begingroup$ That was it! @azerbajdzan, would you post your response as an answer? It took 19s in Mathematica using FindList beating out Terminal and Python by almost 10x $\endgroup$ Jan 18 at 21:11
  • 1
    $\begingroup$ MaxMemoryUsed for FindList gives 40MB for a 3GB file, which is great $\endgroup$
    – jWey
    Jan 19 at 17:55

2 Answers 2

19
$\begingroup$

FindList is meant to deal with such tasks. Not sure whether it works with all types of files but for plaintext files like in the case of CSV file it should work.

$\endgroup$
1
  • 1
    $\begingroup$ FindList (3s) was much faster than grep (33s) or python (70s) for a case insensitive search (30150 out of 3133378 lines) of a 3GB csv. $\endgroup$
    – jWey
    Jan 19 at 11:03
1
$\begingroup$

With python you have GIL, which makes a lot of things run slower.

You could use a library like polars to speed things up. e.g.

import polars as pl
def cell(filepath):
    df = pl.read_csv()
    return df.to_dicts() # turns dataframe into list of dict
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.