5
$\begingroup$

I am working to develop a bioinformatic application that will require the development of a huge hash table with billions of keys.

As a simple example, one key might have the following string sequence: "TGGAC" - which has a ByteCount of 32.

Is it possible in Mathematica to store each letter of the 5-letter string as a 2-bit binary number to reduce the memory requirement for each key?

For example: A might be encoded as 00 T might be encoded as 01 G might be encoded as 10 C might be encoded as 11

$\endgroup$
  • 4
    $\begingroup$ There are many ways you can encode data in Mathematica, for instance as integers: FromDigits[Characters["TGGAC"] /. Thread[{"A", "T", "G", "C"} -> Range[0, 3]], 4] // ByteCount -> (* 16 *). As you can see, short objects are dominated by per-object overheads. There's a possibility to work around this by packing more of them on a single object. $\endgroup$ – kirma Jun 12 at 19:11
  • $\begingroup$ If the strings are not too long there might be a further saving in sorting and then storing them as a packed array. But searching would then be O(log n) instead of O(1). $\endgroup$ – Daniel Lichtblau Jun 13 at 21:40
2
$\begingroup$

Using kirma's advice in the comment section:

Encoding:

   FromDigits[
 Characters["TGGAC"] /. Thread[{"A", "T", "G", "C"} -> Range[0, 3]],
  4]

(* output: 419 *)

Decoding:

    StringJoin[
 Characters[IntegerString[419, 4]] /. 
  Thread[Map[ToString[#] &, Range[0, 3]] -> {"A", "T", "G", "C"}]]

(* output: TGGAC *)

Using this method I was able to lower the original hash table byte count on my data (RNA sequencing) from 69.17 Gb to 27.70 Gb. (Much obliged!)

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.