Return to Revisions

Summary

This is generalized piece of code for reading FAST rows terminated with a newline from a CSV or TSV text stream with UTF8 encoding.

You may specify options for the filetype, the starting and ending line number, as well as a string to search for a position in the file (find).

The import option (True/False) controls whether there is going to be a post-processing of the rows read from the file with the ImportString function. This is slower than the ReadList with the types option or with import->False.

Code

null[_String] := Null
countLines = Length@ReadList[#, null@String, NullRecords -> True] &;

ClearAll[readRows];
Options[readRows]={filetype->"CSV",start->0,end->-1, types->{}, import->True, find->""};
readRows[fname_String?FileExistsQ,OptionsPattern[]]:=Module[
{txtstrm, ftype,typesval,impmethod,
 startLine,endLine,line1,str, 
 hdr, body, bodyImport},

(* Blank function for skipping word separators, e.g. commas *)
blnk[]=blnk[Character];
blnk[_String]=Sequence[];

(* Get Option Values *)
startLine=OptionValue[start];endLine=OptionValue[end];
ftype=OptionValue[filetype];typesval=OptionValue[types];
impmethod=OptionValue[import];str=OptionValue[find];

(* Open Stream *)
txtstrm=OpenRead[fname];

(* Process Header, i.e. first line *)
line1 = ReadList[txtstrm,Record,1][[1]];
Which[
 ftype=="CSV", hdr = StringSplit[line1,","],
 ftype=="TSV", hdr = StringSplit[line1],
 True, Abort[] ];

(* If user has not specified any types for ReadLine, types\[Rule]{} *)
If[Length@typesval==0,typesval= Riffle[Table[Word,Length@hdr],blnk[]]];

(* If user has not specified the line to end reading *)
If[endLine==-1,endLine=countLines@fname];

(* If user specified to start reading the text stream from a specific line *)
If [startLine!=0,Skip[txtstrm,Record,startLine]];

(* If user specified a string to find in a file *)
If[str!="",Find[txtstrm,str]];

(* Check the import option for post-processing each row with ImportString *)
If[impmethod,
 body =ReadList[txtstrm,Record,endLine-startLine]; 
 bodyImport= ImportString[StringJoin[Riffle[body,"\n"]],ftype,CharacterEncoding->"UTF-8"],
(* else *)
body =ReadList[txtstrm,typesval,endLine-startLine,WordSeparators->{"\n",","}]];

(* Close the stream *)
Close[txtstrm];

(* Add the header at the beginning, return the result *)
If[impmethod,Prepend[bodyImport,hdr],Prepend[body,hdr]]   
] (* End of readRows *)

Acknowledgements

Open Source DBAPI Project

I am planning to include this in my DBAPI project together with another extremely useful piece of code for counting lines that I found here. Corrections, ideas, or further enhancements are most welcome.