Importing or reading data in binary format 212

Question

I have a binary data file in format 212, and I want to import the file into Mathematica. How do I read or import binary format 212? I almost have no direct experience of working with any binary format. I would appreciate your help.

Here is the link to the data file: physionet.org/physiobank/database/mitdb/100.dat

The specification of the format is

"Each sample is represented by a 12-bit two’s complement amplitude. The first sample is obtained from the 12 least significant bits of the first byte pair (stored least significant byte first). The second sample is formed from the 4 remaining bits of the first byte pair (which are the 4 high bits of the 12-bit sample) and the next byte (which contains the remaining 8 bits of the second sample). The process is repeated for each successive pair of samples."

From the following link: http://www.physionet.org/physiotools/wag/signal-5.htm#sect9

Answer 1

If I am not mistaken the 212 format is storing two signals and each 3-byte group provides a 12-bit reading of that pair. Importing with Byte isn't too bad for this set:

data = Import["http://physionet.org/physiobank/database/mitdb/100.dat", "Byte"] ~ Partition ~ 3

Then two helpers to parse it:

twosComplement[a_, n_] := If[a < 2^(n - 1), a, a - 2^n]

parse212[{a_, b_, c_}] := twosComplement[#, 12] & /@ {a + BitAnd[b, 16^^0F]*256, 
   BitAnd[b, 16^^F0]*16 + c}

This runs in about 5s on my machine (650k x3 bytes)

Timing[data2 = parse212 /@ data // Transpose;]

Take a look at the beginning of it - you can see the two different signals:

ListPlot[data2[[All, ;; 500]], Joined -> True]

... or look at the values:

data2[[All, ;; 10]]
{{995, 995, 995, 995, 995, 995, 995, 995, 1000, 997}, {1011, 1011, 
  1011, 1011, 1011, 1011, 1011, 1011, 1008, 1008}}

Answer 2

I recently had a similar task decoding 12-bit binary 24-channel electroencephalogram data files and found that bit shifting and masking with BitAnd are the way to go. The specific approach depends on file structure, endianness and channel number. In the case of this 'format 212' binary data file containing two interlaced signals, the following procedure works well:

Read the file as 24-bit unsigned integers:

file= "http://physionet.org/physiobank/database/mitdb/100.dat";
data24 = BinaryReadList[file, "UnsignedInteger24"];  

Because of byte ordering the first 12-bit signal samples (sig1) are taken from bits 13-24 and the second 12-bit signal samples (sig2) are formed by shifting bits 1-8 to the right 16 places and adding bits 9-12 shifted 4 places to the right:

sig1 = BitAnd[data24, 16^^000FFF];
sig2 = BitShiftRight[data24, 16] + BitShiftRight[BitAnd[data24, 16^^00F000], 4];

Let Export and BinaryReadList do the twos' complement:

Export["tmp1", BitShiftLeft[sig1, 4], "UnsignedInteger16"];
Export["tmp2", BitShiftLeft[sig2, 4], "UnsignedInteger16"];
sig1 = BitShiftRight[BinaryReadList["tmp1", "Integer16"], 4];
sig2 = BitShiftRight[BinaryReadList["tmp2", "Integer16"], 4];
DeleteFile["tmp1"]; DeleteFile["tmp2"];

The initial left 4-bit shift makes 16-bit values and the UnsignedInteger16 export writes them preserving sign bits. The (singed) Integer16 BinaryRead then creates the twos' complement with signing as necessary (this file happens to have no negative values). The final right 4-bit shift restores the original magnitudes. Plot:

a = 1; b = 1024;
ListLinePlot[{sig1[[a ;; b]], sig2[[a ;; b]]}, PlotRange -> All, ImageSize->600, AspectRatio -> Automatic]

This procedure's output is identical to SEngstrom's 3-byte algorithm posted above, but decodes the entire file in 0.1 s (OS X 64-bit i7).

Regards,

DBM

p.s. I also wrote a small c++ command line tool 'b12to16cl' that converts this format212 file to 16-bit binary in < 0.0005 s. It can be called from within Mathematica:

Run["/usr/local/bin/b12to16cl format212 inFilePath outFilePath"]

Then one can just BinaryReadList the tmp file as "Integer16" and delete it. Let me know if you'd like the code...

Regards

Answer 3

This might work... however, it is very inefficient.

Slightly faster using Import instead of BinaryReadList:

binary= Partition[Import["http://physionet.org/physiobank/database/mitdb/100.dat", "Bit"], 12];
twoscomplement[bin_] := 
 If[First@bin != 1, 
  FromDigits[Rest@bin, 2], 
  -FromDigits[Rest@bin /. {1 -> 0, 0 -> 1}, 2] - 1
 ]
output = twoscomplement /@ binary;

output // Short

(*    

{-1587, 1011, -1587, 1011, -1587, 1011, <<1299988>>, 1939, 951, 1651, 957, 4, 768}

*)

ListPlot[output[[;; 5000]]]

Answer 4

Another variant..

twoscomplement[bin_] := If[First@bin != 1,
      FromDigits[bin, 2], BitOr[FromDigits[bin, 2], -2^12]]

or to be safe..

twoscomplement[bin_List /; Length[bin] == 12] := If[First@bin != 1,
     FromDigits[bin, 2], BitOr[FromDigits[bin, 2], -2^12]]