General method
Since the emphasis of this question is on performance I propose something like this:
extractColumn[file_, col_] :=
Module[{bin, len},
bin = BinaryReadList @ file;
len = 1 + LengthWhile[bin, # =!= 10 &];
bin[[col ;;;; len]] // FromCharacterCode
]
Or, if you prefer, condensed:
extractColumn[file_, col_] :=
FromCharacterCode @
#[[col ;;;; 1 + LengthWhile[#, # =!= 10 &]]] & @
BinaryReadList @ file
This directly reads in the text file as 8-bit integers; it would be possible to use 16-bit integers if the file uses. 16-bit encoding. It then finds a line return (10) to determine line length, and extracts the column with Part and Span; finally the output string is reconstructed with FromCharacterCode.
To test it I make a 44MB rectangular text file "textblock.txt":
Export["textblock.txt",
Riffle[
StringJoin /@ RandomChoice["a"~CharacterRange~"z", {500000, 90}],
"\n"
] // StringJoin
]
Then call:
extractColumn["textblock.txt", 17] // StringLength // Timing
{0.406, 500000}
Methodology for binary data
In a comment you state:
My text files are files containing only " ", "o", and return
characters ("\r" and/or "\n"). The " " and "o" characters comprise an
"existence matrix," where " " represents False and "o" represents
True. The row number and the column number each have meaning (the row
number represents "type" while the column number represents "time"). I
would like to sequentially read in each column and perform analysis on
each column.
So it would be nice if I could read in multiple columns at once, since
I think BinaryReadList@file may be relatively expensive (my text files
have 200,001 columns each), so calling extractColumn 200,001 times
(each call taking on the order of 1 second) can be quite time
consuming.
Okay, I'll create a new 98MB sample file as follows:
Export["TFblock.txt",
Riffle[StringJoin /@ RandomChoice[{" ", "o"}, {500, 200000}], "\n"] // StringJoin
]
On a modern machine you should be able to read this entire file at once, as strings (using about the same memory):
strings = ReadList["TFblock.txt", Record];
ByteCount[strings]
MaxMemoryUsed[]
100024032
115317104
If for some reason this is not possible, e.g. you start working with huge files, you could use Read[stream, Record] to do this (the following) one row at a time.
The first rule of working efficiently with binary data is: store it in a binary format.
Since Mathematica is equipped to handle very large integers we can store each row of this data as a single Integer. As strings, each data point takes one byte, but packed in an integer each takes only one bit:
strings[[1]] // ByteCount
200040
stringToInt = # ~StringReplace~ {" " -> "0", "o" -> "1"} ~FromDigits~ 2 &;
stringToInt @ strings[[1]] // ByteCount
25040
This operation is a bit slow but with it the entire data is stored in a mere 12.5MB:
Timing[ints = stringToInt /@ StringReverse @ strings;]
ByteCount[ints]
{8.627, Null}
12524032
We can use BitGet to pull data from the Integers, indexed from zero at the least significant bit. (This is the reason for StringReverse above.) This returns the first column: BitGet[ints, 0]. This the last: BitGet[ints, 199999].
The operation is reasonably fast:
Do[BitGet[ints, i], {i, 0, 4999}]; // Timing
{0.421, Null}
Confirmation of result:
StringJoin[BitGet[ints, 16] /. {0 -> " ", 1 -> "o"}] ===
extractColumn["TFblock.txt", 17]
True
Of course if the data is always accessed by column it would be better to have the data in that format to begin with. You could make a one-time pass through the columns using BitGet and FromDigits to transpose the data:
Timing[
ints2 = Table[ints ~BitGet~ i ~FromDigits~ 2, {i, 0, 199999}];
]
{19.391, Null}
A column is then read with: IntegerDigits[ints2[[ (*column*) ]], 2, (* row length *)]. Row length must be specified so as not to lose leading zeros. Confirmation:
IntegerDigits[ints2[[17]], 2, 500] === BitGet[ints, 16]
True
Data extraction is now very fast:
Do[IntegerDigits[ints2[[i]], 2, 500], {i, 1, 5000}] // Timing
{0.016, Null}
"Table"instead of"String", or evenReadList? $\endgroup$