How granular is memory usage for computational purposes in Mathematica?

Question

I am working to develop a bioinformatic application that will require the development of a huge hash table with billions of keys.

As a simple example, one key might have the following string sequence: "TGGAC" - which has a ByteCount of 32.

Is it possible in Mathematica to store each letter of the 5-letter string as a 2-bit binary number to reduce the memory requirement for each key?

For example: A might be encoded as 00 T might be encoded as 01 G might be encoded as 10 C might be encoded as 11

There are many ways you can encode data in Mathematica, for instance as integers: FromDigits[Characters["TGGAC"] /. Thread[{"A", "T", "G", "C"} -> Range[0, 3]], 4] // ByteCount -> (* 16 *). As you can see, short objects are dominated by per-object overheads. There's a possibility to work around this by packing more of them on a single object. — kirma, Commented Jun 12, 2020 at 19:11
If the strings are not too long there might be a further saving in sorting and then storing them as a packed array. But searching would then be O(log n) instead of O(1). — Daniel Lichtblau, Commented Jun 13, 2020 at 21:40

Todd Allen · Accepted Answer · 2020-06-23 00:55:56Z

2

Using kirma's advice in the comment section:

Encoding:

   FromDigits[
 Characters["TGGAC"] /. Thread[{"A", "T", "G", "C"} -> Range[0, 3]],
  4]

(* output: 419 *)

Decoding:

    StringJoin[
 Characters[IntegerString[419, 4]] /. 
  Thread[Map[ToString[#] &, Range[0, 3]] -> {"A", "T", "G", "C"}]]

(* output: TGGAC *)

Using this method I was able to lower the original hash table byte count on my data (RNA sequencing) from 69.17 Gb to 27.70 Gb. (Much obliged!)

answered Jun 23, 2020 at 0:55

Todd Allen

2,1345 gold badges29 silver badges32 bronze badges

Add a comment |

1 Answer 1