# How granular is memory usage for computational purposes in Mathematica?

I am working to develop a bioinformatic application that will require the development of a huge hash table with billions of keys.

As a simple example, one key might have the following string sequence: "TGGAC" - which has a ByteCount of 32.

Is it possible in Mathematica to store each letter of the 5-letter string as a 2-bit binary number to reduce the memory requirement for each key?

For example: A might be encoded as 00 T might be encoded as 01 G might be encoded as 10 C might be encoded as 11

• There are many ways you can encode data in Mathematica, for instance as integers: FromDigits[Characters["TGGAC"] /. Thread[{"A", "T", "G", "C"} -> Range[0, 3]], 4] // ByteCount -> (* 16 *). As you can see, short objects are dominated by per-object overheads. There's a possibility to work around this by packing more of them on a single object. Commented Jun 12, 2020 at 19:11
• If the strings are not too long there might be a further saving in sorting and then storing them as a packed array. But searching would then be O(log n) instead of O(1). Commented Jun 13, 2020 at 21:40

Using kirma's advice in the comment section:

Encoding:

   FromDigits[
Characters["TGGAC"] /. Thread[{"A", "T", "G", "C"} -> Range[0, 3]],
4]


(* output: 419 *)

Decoding:

    StringJoin[
Characters[IntegerString[419, 4]] /.
Thread[Map[ToString[#] &, Range[0, 3]] -> {"A", "T", "G", "C"}]]


(* output: TGGAC *)

Using this method I was able to lower the original hash table byte count on my data (RNA sequencing) from 69.17 Gb to 27.70 Gb. (Much obliged!)