10
$\begingroup$

The list of character encodings currently supported by Mathematica is still extremely limited. For example, it supports only three standard IBM PC (OEM) code pages:

Select[$CharacterEncodings, StringMatchQ[#, "CP" ~~ __] &]
{"CP936", "CP949", "CP950"}

Java already has built-in support for many standard code pages including code page 866 which I'm currently interested in.

Here is an example text as a "Byte" sequence originally encoded using code page 866 which can easily be obtained on Russian-localized Windows with the following command:

i = Import["!help", "Byte"]
Export["i.txt", i]

From the above link the "Byte" sequence can be restored using the command:

Import["http://pastebin.com/raw/BtTxh0U6", "List"] == i
True

How to decode in Mathematica a String (or a "Byte" sequence) encoded using code page 866?


P.S. This answer and this thread contain potentially useful pieces of code in Java. From the second link it seems that I should use either Charset or CharsetDecoder class but I can't figure out how to continue from the following:

Needs["JLink`"];
InstallJava[];

charsetDecoderClass = LoadJavaClass["java.nio.charset.CharsetDecoder"];

charsetClass = LoadJavaClass["java.nio.charset.Charset"];
$\endgroup$
3
$\begingroup$

UPDATE 3

Probably the most elegant complete solution (output is truncated):

Needs["JLink`"];
InstallJava[];

ImportString[
 JavaBlock[JavaObjectToExpression[JavaNew["java.lang.String", i, "CP866"]]], "Text"]
"Для получения сведений об определенной команде наберите HELP <имя команды>
ASSOC          Вывод либо изменение сопоставлений по расширениям имен файлов.
ATTRIB         Отображение и изменение атрибутов файлов.
BREAK        Включение и выключение режима обработки комбинации клавиш CTRL+C.
BCDEDIT        Задает свойства в базе данных загрузки для управления начальной
               загрузкой.
CACLS          Отображение и редактирование списков управления доступом (ACL)
               к файлам.

Packaged as a function:

Needs["JLink`"];
InstallJava[];

decodeCP866[b : {__Integer}] := 
  ImportString[JavaBlock[JavaObjectToExpression[JavaNew["java.lang.String", b, "CP866"]]],
    "Text"];
decodeCP866[s_String] := decodeCP866[ImportString[s, "Byte"]]

UPDATE 2

A bit ugly method to get rid of unwanted "\r" in the output using Java only (output is truncated):

Needs["JLink`"];
InstallJava[];

JavaBlock[JavaNew["java.lang.String", i, "CP866"]@
  replace[MakeJavaObject@"\r\n", MakeJavaObject@"\n"]]
"Для получения сведений об определенной команде наберите HELP <имя команды>
ASSOC          Вывод либо изменение сопоставлений по расширениям имен файлов.
ATTRIB         Отображение и изменение атрибутов файлов.
BREAK        Включение и выключение режима обработки комбинации клавиш CTRL+C.
BCDEDIT        Задает свойства в базе данных загрузки для управления начальной
               загрузкой.

Two alternatives using StringReplace:

StringReplace[JavaBlock[JavaObjectToExpression[JavaNew["java.lang.String", i, "CP866"]]], 
 "\r\n" -> "\n"]

or

StringReplace[JavaBlock[JavaNew["java.lang.String", i, "CP866"]@toString[]], 
 "\r\n" -> "\n"]

However it would be much nicer to be able to switch off the addition of "\r" by Java.


UPDATE 1

Actually the problem can be solved as simple as

Needs["JLink`"];
InstallJava[];

JavaBlock[JavaObjectToExpression[JavaNew["java.lang.String", i, "CP866"]]]

No need to load any classes explicitly and it is sufficient to pass the name of charset as String!

But I still haven't found a way to avoid conversion of "\n" into "\r\n\". The following piece of code has no effect:

sys = LoadJavaClass@"java.lang.System";
sys@setProperty["line.separator", "\n"]

Original answer

Loading the Charset JavaClass:

Needs["JLink`"];
InstallJava[];

LoadJavaClass["java.nio.charset.Charset"]
JavaClass[java.nio.charset.Charset,<>]

Creating JavaObject corresponding to the CP866 charset:

cp866CharsetObject = Charset`forName["CP866"]
«JavaObject[sun.nio.cs.IBM866]»

Converting a "Byte" sequence into a String JavaObject, then converting this object into Mathematica expression and releasing this JavaObject (output is truncated):

JavaBlock[JavaObjectToExpression[JavaNew["java.lang.String", i, cp866CharsetObject]]]
"Для получения сведений об определенной команде наберите HELP <имя команды>

ASSOC          Вывод либо изменение сопоставлений по расширениям имен файлов.

ATTRIB         Отображение и изменение атрибутов файлов.

BREAK        Включение и выключение режима обработки комбинации клавиш CTRL+C.

BCDEDIT        Задает свойства в базе данных загрузки для управления начальной

               загрузкой.

The only problem with this solution is that it decodes the newline characters as "\r\n\" what Mathematica interprets as two newline characters instead of one.

$\endgroup$
  • $\begingroup$ Here is an alternative solution by modifying the registry default settings for CMD. With this one can simply use Import["!help", "Text", CharacterEncoding -> "WindowsCyrillic"] on Russian-localized Windows. $\endgroup$ – Alexey Popkov Feb 25 '16 at 17:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.