Bounty Ended with 50 reputation awarded by Andrew
4 added 3358 characters in body; deleted 1 characters in body
source | link

General method

{0.406, 500000}

Methodology for binary data

In a comment you state:

My text files are files containing only " ", "o", and return characters ("\r" and/or "\n"). The " " and "o" characters comprise an "existence matrix," where " " represents False and "o" represents True. The row number and the column number each have meaning (the row number represents "type" while the column number represents "time"). I would like to sequentially read in each column and perform analysis on each column.

So it would be nice if I could read in multiple columns at once, since I think BinaryReadList@file may be relatively expensive (my text files have 200,001 columns each), so calling extractColumn 200,001 times (each call taking on the order of 1 second) can be quite time consuming.

Okay, I'll create a new 98MB sample file as follows:

Export["TFblock.txt", 
  Riffle[StringJoin /@ RandomChoice[{" ", "o"}, {500, 200000}], "\n"] // StringJoin
]

On a modern machine you should be able to read this entire file at once, as strings (using about the same memory):

strings = ReadList["TFblock.txt", Record];

ByteCount[strings]

MaxMemoryUsed[]

100024032

115317104

If for some reason this is not possible, e.g. you start working with huge files, you could use Read[stream, Record] to do this (the following) one row at a time.

The first rule of working efficiently with binary data is: store it in a binary format.

Since Mathematica is equipped to handle very large integers we can store each row of this data as a single Integer. As strings, each data point takes one byte, but packed in an integer each takes only one bit:

strings[[1]] // ByteCount

200040

stringToInt = # ~StringReplace~ {" " -> "0", "o" -> "1"} ~FromDigits~ 2 &;

stringToInt @ strings[[1]] // ByteCount

25040

This operation is a bit slow but with it the entire data is stored in a mere 12.5MB:

Timing[ints = stringToInt /@ StringReverse @ strings;]

ByteCount[ints]

{8.627, Null}

12524032

We can use BitGet to pull data from the Integers, indexed from zero at the least significant bit. (This is the reason for StringReverse above.) This returns the first column: BitGet[ints, 0]. This the last: BitGet[ints, 199999].

The operation is reasonably fast:

Do[BitGet[ints, i], {i, 0, 4999}]; // Timing

{0.421, Null}

Confirmation of result:

StringJoin[BitGet[ints, 16] /. {0 -> " ", 1 -> "o"}] === 
 extractColumn["TFblock.txt", 17]

True

Of course if the data is always accessed by column it would be better to have the data in that format to begin with. You could make a one-time pass through the columns using BitGet and FromDigits to transpose the data:

Timing[
 ints2 = Table[ints ~BitGet~ i ~FromDigits~ 2, {i, 0, 199999}];
]

{19.391, Null}

A column is then read with: IntegerDigits[ints2[[ (*column*) ]], 2, (* row length *)]. Row length must be specified so as not to lose leading zeros. Confirmation:

IntegerDigits[ints2[[17]], 2, 500] === BitGet[ints, 16]

True

Data extraction is now very fast:

Do[IntegerDigits[ints2[[i]], 2, 500], {i, 1, 5000}] // Timing

{0.016, Null}

{0.406, 500000}

General method

{0.406, 500000}

Methodology for binary data

In a comment you state:

My text files are files containing only " ", "o", and return characters ("\r" and/or "\n"). The " " and "o" characters comprise an "existence matrix," where " " represents False and "o" represents True. The row number and the column number each have meaning (the row number represents "type" while the column number represents "time"). I would like to sequentially read in each column and perform analysis on each column.

So it would be nice if I could read in multiple columns at once, since I think BinaryReadList@file may be relatively expensive (my text files have 200,001 columns each), so calling extractColumn 200,001 times (each call taking on the order of 1 second) can be quite time consuming.

Okay, I'll create a new 98MB sample file as follows:

Export["TFblock.txt", 
  Riffle[StringJoin /@ RandomChoice[{" ", "o"}, {500, 200000}], "\n"] // StringJoin
]

On a modern machine you should be able to read this entire file at once, as strings (using about the same memory):

strings = ReadList["TFblock.txt", Record];

ByteCount[strings]

MaxMemoryUsed[]

100024032

115317104

If for some reason this is not possible, e.g. you start working with huge files, you could use Read[stream, Record] to do this (the following) one row at a time.

The first rule of working efficiently with binary data is: store it in a binary format.

Since Mathematica is equipped to handle very large integers we can store each row of this data as a single Integer. As strings, each data point takes one byte, but packed in an integer each takes only one bit:

strings[[1]] // ByteCount

200040

stringToInt = # ~StringReplace~ {" " -> "0", "o" -> "1"} ~FromDigits~ 2 &;

stringToInt @ strings[[1]] // ByteCount

25040

This operation is a bit slow but with it the entire data is stored in a mere 12.5MB:

Timing[ints = stringToInt /@ StringReverse @ strings;]

ByteCount[ints]

{8.627, Null}

12524032

We can use BitGet to pull data from the Integers, indexed from zero at the least significant bit. (This is the reason for StringReverse above.) This returns the first column: BitGet[ints, 0]. This the last: BitGet[ints, 199999].

The operation is reasonably fast:

Do[BitGet[ints, i], {i, 0, 4999}]; // Timing

{0.421, Null}

Confirmation of result:

StringJoin[BitGet[ints, 16] /. {0 -> " ", 1 -> "o"}] === 
 extractColumn["TFblock.txt", 17]

True

Of course if the data is always accessed by column it would be better to have the data in that format to begin with. You could make a one-time pass through the columns using BitGet and FromDigits to transpose the data:

Timing[
 ints2 = Table[ints ~BitGet~ i ~FromDigits~ 2, {i, 0, 199999}];
]

{19.391, Null}

A column is then read with: IntegerDigits[ints2[[ (*column*) ]], 2, (* row length *)]. Row length must be specified so as not to lose leading zeros. Confirmation:

IntegerDigits[ints2[[17]], 2, 500] === BitGet[ints, 16]

True

Data extraction is now very fast:

Do[IntegerDigits[ints2[[i]], 2, 500], {i, 1, 5000}] // Timing

{0.016, Null}

3 added 4 characters in body; deleted 1 characters in body
source | link

Since the emphasis of this question is on performance I propose something like this:

extractColumn[file_, col_] :=
 Module[{bin, len},
   bin = BinaryReadList @ file;
   len = 1 + LengthWhile[bin, # =!= 10 &];
   bin[[7bin[[col ;;;; len]] // FromCharacterCode
 ]

Or, if you prefer, condensed:

extractColumn[file_, col_] :=
  FromCharacterCode @
    #[[7 ;;#[[col ;;;;;; 1 + LengthWhile[#, # =!= 10 &]]] & @ 
      BinaryReadList @ file

This directly reads in the text file as 8-bit integers; it would be possible to use 16-bit integers if the file uses. 16-bit encoding. It then finds a line return (10) to determine line length, and extracts the column with Part and Span; finally the output string is reconstructed with FromCharacterCode.

To test it I make a 44MB rectangular text file "textblock.txt":

Export["textblock.txt",
 Riffle[
   StringJoin /@ RandomChoice["a"~CharacterRange~"z", {500000, 90}],
   "\n"
 ] // StringJoin
]

Then call:

extractColumn["textblock.txt", 17] // StringLength // Timing
{0.406, 500000}

Since the emphasis of this question is on performance I propose something like this:

extractColumn[file_, col_] :=
 Module[{bin, len},
   bin = BinaryReadList @ file;
   len = 1 + LengthWhile[bin, # =!= 10 &];
   bin[[7 ;;;; len]] // FromCharacterCode
 ]

Or, if you prefer, condensed:

extractColumn[file_, col_] :=
  FromCharacterCode @
    #[[7 ;; ;; 1 + LengthWhile[#, # =!= 10 &]]] & @ 
      BinaryReadList @ file

This directly reads in the text file as 8-bit integers; it would be possible to use 16-bit integers if the file uses. 16-bit encoding. It then finds a line return (10) to determine line length, and extracts the column with Part and Span; finally the output string is reconstructed with FromCharacterCode.

To test it I make a 44MB rectangular text file "textblock.txt":

Export["textblock.txt",
 Riffle[
   StringJoin /@ RandomChoice["a"~CharacterRange~"z", {500000, 90}],
   "\n"
 ] // StringJoin
]

Then call:

extractColumn["textblock.txt", 17] // StringLength // Timing
{0.406, 500000}

Since the emphasis of this question is on performance I propose something like this:

extractColumn[file_, col_] :=
 Module[{bin, len},
   bin = BinaryReadList @ file;
   len = 1 + LengthWhile[bin, # =!= 10 &];
   bin[[col ;;;; len]] // FromCharacterCode
 ]

Or, if you prefer, condensed:

extractColumn[file_, col_] :=
  FromCharacterCode @
    #[[col ;;;; 1 + LengthWhile[#, # =!= 10 &]]] & @ 
      BinaryReadList @ file

This directly reads in the text file as 8-bit integers; it would be possible to use 16-bit integers if the file uses. 16-bit encoding. It then finds a line return (10) to determine line length, and extracts the column with Part and Span; finally the output string is reconstructed with FromCharacterCode.

To test it I make a 44MB rectangular text file "textblock.txt":

Export["textblock.txt",
 Riffle[
   StringJoin /@ RandomChoice["a"~CharacterRange~"z", {500000, 90}],
   "\n"
 ] // StringJoin
]

Then call:

extractColumn["textblock.txt", 17] // StringLength // Timing
{0.406, 500000}
2 added 187 characters in body; added 1 characters in body
source | link

Since the emphasis of this question is on performance I propose something like this:

extractColumn[file_, col_] :=
 Module[{bin, len},
   bin = BinaryReadList @ file;
   len = 1 + LengthWhile[bin, # =!= 10 &];
   bin[[7 ;;;; len]] // FromCharacterCode
 ]

Or, if you prefer, condensed:

extractColumn[file_, col_] :=
  FromCharacterCode @
    #[[7 ;; ;; 1 + LengthWhile[#, # =!= 10 &]]] & @ 
      BinaryReadList @ file

This directly reads in the text file as 8-bit integers; it would be possible to use 16-bit integers if the file uses. 16-bit encoding. It then finds a line return (10) to determine line length, and extracts the column with Part and Span; finally the output string is reconstructed with FromCharacterCode.

To test it I make a 44MB rectangular text file "textblock.txt":

Export["textblock.txt",
 Riffle[
   StringJoin /@ RandomChoice["a"~CharacterRange~"z", {500000, 90}],
   "\n"
 ] // StringJoin
]

Then call:

extractColumn["textblock.txt", 17] // StringLength // Timing
{0.406, 500000}

Since the emphasis of this question is on performance I propose something like this:

extractColumn[file_, col_] :=
 Module[{bin, len},
   bin = BinaryReadList @ file;
   len = 1 + LengthWhile[bin, # =!= 10 &];
   bin[[7 ;;;; len]] // FromCharacterCode
 ]

This directly reads in the text file as 8-bit integers; it would be possible to use 16-bit integers if the file uses. 16-bit encoding. It then finds a line return (10) to determine line length, and extracts the column with Part and Span; finally the output string is reconstructed with FromCharacterCode.

To test it I make a 44MB rectangular text file "textblock.txt":

Export["textblock.txt",
 Riffle[
   StringJoin /@ RandomChoice["a"~CharacterRange~"z", {500000, 90}],
   "\n"
 ] // StringJoin
]

Then call:

extractColumn["textblock.txt", 17] // StringLength // Timing
{0.406, 500000}

Since the emphasis of this question is on performance I propose something like this:

extractColumn[file_, col_] :=
 Module[{bin, len},
   bin = BinaryReadList @ file;
   len = 1 + LengthWhile[bin, # =!= 10 &];
   bin[[7 ;;;; len]] // FromCharacterCode
 ]

Or, if you prefer, condensed:

extractColumn[file_, col_] :=
  FromCharacterCode @
    #[[7 ;; ;; 1 + LengthWhile[#, # =!= 10 &]]] & @ 
      BinaryReadList @ file

This directly reads in the text file as 8-bit integers; it would be possible to use 16-bit integers if the file uses. 16-bit encoding. It then finds a line return (10) to determine line length, and extracts the column with Part and Span; finally the output string is reconstructed with FromCharacterCode.

To test it I make a 44MB rectangular text file "textblock.txt":

Export["textblock.txt",
 Riffle[
   StringJoin /@ RandomChoice["a"~CharacterRange~"z", {500000, 90}],
   "\n"
 ] // StringJoin
]

Then call:

extractColumn["textblock.txt", 17] // StringLength // Timing
{0.406, 500000}
1
source | link