1
$\begingroup$

I am struggling to convert a base64 string to a list of UnsignedInteger16 values. I have limited experience with mathematica, so please excuse me if this should be obvious.

I read the base64 string from an XML file and ultimately into a variable base64String. This is a long string with 7057 UnsignedInteger16 values, but it starts off as: "8ynnKdspzinCKbcprSmhKZUpiCl8KW8pYylXKUkpQCk0KSop..."

Can convert to ByteArry with n = BaseDecode[base64String] // Normal;

Then n has the expected values {243, 41, 231, 41, 219, 41, 206, 41, 194, 41, 183, 41, 173, 41, 161, ...

However, what I am looking for is a list with the values {243 + 256 * 41, 231 + 256 * 41, 219 + 256 * 41, ...

i.e.: {10739, 10727, 10727, ...

Thank you in advance for you suggestions.

Edit: Thank you for the suggestion to use Partition. Works well. For the interests in improving my understanding, can any one suggest why I cannot get ImportString[...] to work.

My source data is XML file with data along the lines of the following where I truncated the Base64 string for clarity.:

<?xml version="1.0" encoding="utf-8" ?>
<BPplus version="5.0" filename="BPplus_00100.xml">
    <MeasDataLogger guid="ee7bee36-ffd6-30ae-53f1-257f0eab7ddd">
        <Sys>130</Sys>
        <Dia>77</Dia>
        <Map>101</Map>
        <PressureWaves>
            <RawPressureWave>
                <RawPressure>NIBP</RawPressure>
                <RawSampleCount>7057</RawSampleCount>
                <RawCuffPressureWave>8ynnKdspzinCKbcprSmhKZUpiCl8KW8pYylXKUk... bU1mjV+NWM1RjU=</RawCuffPressureWave>
            </RawPressureWave>
        </PressureWaves>
    </MeasDataLogger>
</BPplus>

The script I then run with the C1, C2 & C3 suggestions.

sampleFilenameString = "C:\\BPPdata\\BPplus_00112.xml";
xmldata = Import[sampleFilenameString, "XML"];
RawCuffPressureWave = 
  Cases[xmldata, XMLElement["RawCuffPressureWave", _, _], Infinity];
nibpxml = RawCuffPressureWave[[2]];
base64String = ToString[nibpxml[[3]]];
C1 = Partition[Normal@BaseDecode[base64String], 2].{1, 256};
C2 = ImportByteArray[BaseDecode[base64String], "UnsignedInteger16"];
C3 = ImportString[base64String, {"Base64", "UnsignedInteger16"}, 
  ByteOrdering -> -1]
C1 == C2
C1 == C3
C1

produces the following output

{17956, 26977, 25964}
True
False
{10739, 10727, 10715, 10702, 10690, 10679, 10669, 10657, ... }

I have tried to make the first parameter ToString[base64String], but that did not change the outcome.

If I manually define the string $base64 = "8ynnKdspzinCKbcprSmhKZUpiCl8KW8pYylXKUkpQCk0KSo....

It works as expected and C1 = C4

C4 = ImportString[$base64, {"Base64", "UnsignedInteger16"}, 
   ByteOrdering -> -1];
C1 == C4

Can anyone explain the difference why $base64 variable works but base64String does not?

Edit: base64String is not a string! Not sure if this is the best way to convert, but it works in Mathematica 11.x and higher.

sampleFilenameString = "C:\\BPPdata\\BPplus_00112.xml";
xmldata = Import[sampleFilenameString, "XML"];
RawCuffPressureWave = Cases[xmldata, XMLElement["RawCuffPressureWave", _, _], Infinity];
nibpxml = RawCuffPressureWave[[2]];
(* convert to string then base64 decode *)
base64Data = 
  Developer`DecodeBase64ToByteArray[
     nibpxml //. XMLElement[_, _, t_] :> t // Flatten // StringJoin]  // Normal;
C1 = Partition[base64Data, 2].{1, 256};
$\endgroup$
5
  • 2
    $\begingroup$ ImportByteArray[BaseDecode[str], "UnsignedInteger16"] ? $\endgroup$
    – ilian
    Jul 20, 2020 at 15:31
  • $\begingroup$ Or Partition[n, 2].{1, 256} $\endgroup$
    – Roman
    Jul 20, 2020 at 15:42
  • $\begingroup$ Looks like we need to use something along the lines of the following to convert the XmlElement data to Text (* convert to string then base64 decode *) base64Data = Developer`DecodeBase64ToByteArray[ ToString[ nibpxml //. XMLElement[_, _, t_] :> t // Flatten // StringJoin]] // Normal; C1 = Partition[base64Data, 2].{1, 256}; $\endgroup$ Jul 21, 2020 at 2:49
  • $\begingroup$ The issue is that the variable base64String in the question contains extraneous braces, i.e. "{8ynnKdsp...}". This is not a valid Base64 string -- but apparently BaseDecode will ignore such errors and process it anyway. Note how the content portion of the XML element is actually a list: XMLElement["RawCuffPressureWave", {}, {"8ynnKdsp..."}]. The call to ToString converts the entire list to a string, braces and all. Assuming the content of the XML element RawCuffPressureWave is always a single text node then the simplest fix is base64String = nibpxml[[3, 1]]. $\endgroup$
    – WReach
    Jul 21, 2020 at 5:45
  • $\begingroup$ thank you for helping improve my understanding re nibpxml[[3]] being a list and the need to index to the first item to get the value with nibpxml[[3,1]]. Probably a bit obvious in retrospect, but eluded me a few days ago. I will remember! :-) $\endgroup$ Jul 23, 2020 at 11:09

2 Answers 2

1
$\begingroup$

We can use ImportString to decode the Base64 string and interpret the resulting bytes in a single step:

ImportString["8ynnKdsp", {"Base64", "UnsignedInteger16"}, ByteOrdering -> -1]
(* {10739, 10727, 10715} *)

Explanation

The documentation for the Base64 import format tells us:

The import format and file encoding can be explicitly specified with
Import["file", {"Base64", "format", elems1, elems2, ...].

In this case, format can be "Binary" which is documented to support many data representation elements including "UnsignedInteger16".

Putting all this together with ImportString, we can perform the required conversion:

$base64 = "8ynnKdspzinCKbcprSmhKZUpiCl8KW8pYylXKUkpQCk0KSop";

ImportString[$base64, {"Base64", "Binary", "UnsignedInteger16"}, ByteOrdering -> -1]

(* {10739, 10727, 10715, 10702, 10690, 10679, 10669, 10657, 10645,
    10632, 10620, 10607, 10595, 10583, 10569, 10560, 10548, 10538} *)

The documentation for Binary also tells us:

For any data representation element type, Import["file", type] can be used as a shorthand for Import["file", {"Binary", type}].

So we can use this shorthand form to get the same result:

ImportString[$base64, {"Base64", "UnsignedInteger16"}, ByteOrdering -> -1]

The ByteOrdering option ensures that the unsigned integers are interpreted in "little-endian" format (low-byte first). This option can be omitted on a little-endian machine since it will be the default $ByteOrdering. But even so it does not hurt to include it defensively if there is any possibility that our code might some day find its way onto a big-endian machine.

$\endgroup$
1
  • $\begingroup$ Thank you for the detailed answer.. I am missing some understanding about strings as this approach only works if the string is hard coded into the script. When i read the "string" from an XML file, then it does not work (but the C1 approach with Partition does work!) See question in answer to @Roman $\endgroup$ Jul 21, 2020 at 1:05
1
$\begingroup$

There are many ways of doing this. The fastest I can think of is

Partition[Normal@BaseDecode[str], 2].{1, 256}

which takes less than half of the time that ImportString requires.

Timing comparisons:

A = RandomInteger[255, 10^8] // ByteArray;
S = BaseEncode[A];

C1 = Partition[Normal@BaseDecode[S], 2].{1, 256}; // AbsoluteTiming // First
(*    4.11495    *)

C2 = ImportByteArray[BaseDecode[S], "UnsignedInteger16"]; // AbsoluteTiming // First
(*    5.98029    *)    (*    @ilian    *)

C3 = ImportString[S, {"Base64", "UnsignedInteger16"}, ByteOrdering -> -1]; // AbsoluteTiming // First
(*    9.64049    *)    (*    @WReach    *)

C1 == C2 == C3
(*    True    *)
$\endgroup$
2
  • $\begingroup$ excellent answer with the performance metrics. i will ultimately be processing many of these files, so fast is helpful. I did try the C2 and C3 approached, but I could not get them to work and they still don't work for me, but the C1 approach with Partition works perfectly (nice it's also the fastest :-). There must be something I don't understand about stings and reading values from XML files. $\endgroup$ Jul 21, 2020 at 1:13
  • $\begingroup$ Actually C1 and C2 work, but I cannot get C3 to work.... I will edit question so we can all learn. $\endgroup$ Jul 21, 2020 at 1:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.