How to workaround failures with Unicode filepaths?

Question

^{(Cross-posted on the Wolfram Community, reported to the support as [CASE:3965891].)}

The Mathematica's Kernel and FrontEnd currently work well with Unicode file/directory paths, but some other components of the system contain long-standing bugs which are source of troubles for the users, especially for the users from non-English-speaking countries.

The most recent version of Mathematica 12.1 still fails to Import a PDF file when its path contains non-ASCII characters: under Windows Import returns $Failed, under OSX it returns empty list. This is due to a long-standing bug in the component "PDF.exe" which is responsible for importing of PDF files:

Export["Тест.pdf", ""]
Import[%]

"Тест.pdf"    
$Failed

The same is true for Importing Mathematica's native NB files as "Plaintext" due to a similar long-standing bug in "NBImport.exe":

Export["Тест.nb", ""]
Import[%, {"NB", "Plaintext"}]

"Тест.nb"
$Failed

The new in version 11 HTTPRequest/URLRead functionality also suffer from this bug. Here is an attempt to upload an image with non-ASCII filename to imgur.com using the method from this answer:

Export["Тест.png", Plot[Sin[x], {x, 0, Pi}]]

URLRead[HTTPRequest[
  "http://stackoverflow.com/upload/image", <|
   "Body" -> {"image" -> <|"Content" -> File[%], "MIMEType" -> "image/png"|>}|>]]

And undoubtedly there are other components suffering from this bug because reports about problems with Unicode filepaths keep appearing on this site.

Hence it is worth to have a dedicated thread with a collection of general techniques allowing to workaround such problems. This thread is intended exactly for this purpose. Some guidelines:

• When posting an OS-specific workaround, please include information about OS.
• If a workaround is limited to local file paths and doesn't work for network paths, please mention this.
• Each answer should contain elaborated description of only one general method along with its limitations.

---

Related questions:

• Cannot import Excel (xls or xlsx) files which have non-ASCII characters in the name
• 11.3 - import of files with paths containing non-ASCII characters
• How to deal with Chinese Windows path/filename?
• Can't get Mathematica to Import PDF files with diacritics on the file name
• Mathematica 10 doesn't support path containing Chinese characters
• Bug in handling backslashed Cyrillic symbols in v.10.0.0?
• Problem with my user name
• Wrong function of NotebookDirectory[] in the 12.1 version

Answer 1

Windows-specific: use short paths (short filenames, 8.3 file names)

This method should work with all parts of the Mathematica system.

Limitations

• short path can be found only for existing file/directory

• short paths exist only if generation of them isn't disabled by administrator (by default it is enabled)

• may not work for network paths

• this method isn't reliable for localized versions of Windows because on them the short paths allowed to contain non-ASCII characters (example).

Solutions

There is an undocumented function FileInformation which allows to obtain the short path (works at least starting from Mathematica 11.3, thanks to GenericAccountName):

Export["Тест.txt", ""]
FileInformation["Тест.txt", "AbsoluteShortFileName"]

"Тест.txt"

"D:\\DOCUME~1\\47BB~1.TXT"

Also starting from version 12.0.0 there is an under-documented (but "official") way to obtain the short path:

Information[File["Тест.txt"], "Properties"] // Shallow

{"ObjectType", "FileName", "DirectoryName", "AbsoluteFileName", "AbsoluteShortFileName", "ShortFileName", "FileBaseName", "FileExtension", "CreationDate", "LastAccessDate", <<33>>}

Information[File["Тест.txt"], "AbsoluteShortFileName"]

"D:\\DOCUME~1\\47BB~1.TXT"

Here is a function which expands a file/directory name and then obtains the corresponding short path using cmd.exe by this method (tested on Windows 7 x64 but should work at least starting from Windows XP):

shortPath[longPath_String] /; FileExistsQ[longPath] := 
 Import["!cmd /c for %A in (\"" <> ExpandFileName[longPath] <> "\") do @echo %~sA", "Text"]

Another solution via NETLink` can be found in this great answer by WReach (see also this answer).

Usage

Create a PDF file with Unicode filepath and find the short path to it:

Export["Тест.pdf", Plot[Sin[x], {x, 0, Pi}]]
shortPath["Тест.pdf"]

"Тест.pdf"
"D:\\DOCUME~1\\FF12~1.PDF"

Import this file:

Import[shortPath["Тест.pdf"]]

Answer 2

Import as "String" first, and then apply ImportString or ...

Currently it is the most elegant workaround, and besides that it is expected to be OS-independent.

Limitations

• This method is intended exclusively for import-related tasks.

• If the path defined via $TemporaryDirectory contains non-ASCII characters, ImportString will fail because under the hood it creates a temp file in $TemporaryDirectory and then passes its path to the appropriate importer, hence we return back to the situation with non-ASCII file path (checked with Mathematica 8.0.4 and 11.1.0 on Windows 7 x64). A workaround is to Import from a stream with BinaryFormat -> True (see below).

Solution

It is as simple as described in the head of this answer. You Import your file with Unicode path as "String" (what works!), and then do with the obtained string what you had to do with the original file. For example pass it to ImportString if your purpose was to import the file:

ImportString[Import[<path to PDF file>, "String"], "PDF"]

... or directly include it into HTTPRequest instead of symbolic link to the local file (see the corresponding example in the question):

HTTPRequest["http://stackoverflow.com/upload/image", <|
  "Body" -> {"image" -> <|"Content" -> Import["Тест.png", "String"], 
                          "MIMEType" -> "image/png"|>}|>]

With Mathematica 10 or higher instead of Import one can use ReadString what gives identical result and also is a bit faster:

ImportString[ReadString[<path to PDF file>], "PDF"]

A low-level alternative is to OpenRead the file with option BinaryFormat -> True and then Read it as a whole with the RecordSeparators -> {} option (don't forget to Close it afterwards!):

# &[Read[#, Record, RecordSeparators -> {}], Close[#]] &@
 OpenRead[fileName, BinaryFormat -> True]

... or directly Import from the stream:

# &[Import[#, "PDF"], Close[#]] &@OpenRead[<path to PDF file>, BinaryFormat -> True]

Usage

Import a PDF file with Unicode filepath:

Export["Тест.pdf", Plot[Sin[x], {x, 0, Pi}]]

ImportString[Import["Тест.pdf", "String"], "PDF"]

"Тест.pdf"

Upload an image with Unicode filepath to imgur.com:

Export["Тест.png", Plot[Sin[x], {x, 0, Pi}]]

URLRead[HTTPRequest[
  "http://stackoverflow.com/upload/image", <|
   "Body" -> {"image" -> <|"Content" -> Import["Тест.png", "String"], 
       "MIMEType" -> "image/png"|>}|>]]

"Тест.png"

Answer 3

Works on Linux: set the environment variable LANG to en_US.UTF-8

In particular, with Mathematica installed in ~/opt/Mathematica/11.2.0, running it in the following way appears to make Import and Export work without any special tricks:

LANG=en_US.UTF-8 ~/opt/Mathematica/11.2.0/Executables/Mathematica

When it's run as in the example above, the following commands work as intended:

Export["/tmp/тест.txt", "Hello"]
Import@%

"/tmp/тест.txt"

"Hello"