15
$\begingroup$

(Cross-posted on the Wolfram Community, reported to the support as [CASE:3965891].)

The Mathematica's Kernel and FrontEnd currently work well with Unicode file/directory paths, but some other components of the system contain long-standing bugs which are source of troubles for the users, especially for the users from non-English-speaking countries.

The most recent version of Mathematica 12.0 still fails to Import a PDF file when its path contains non-ASCII characters: under Windows Import returns $Failed, under OSX it returns empty list. This is due to a long-standing bug in the component "PDF.exe" which is responsible for importing of PDF files:

Export["Тест.pdf", ""]
Import[%]
"Тест.pdf"    
$Failed

The same is true for Importing Mathematica's native NB files as "Plaintext" due to a similar long-standing bug in "NBImport.exe":

Export["Тест.nb", ""]
Import[%, {"NB", "Plaintext"}]
"Тест.nb"
$Failed

The new in version 11 HTTPRequest/URLRead functionality also suffer from this bug. Here is an attempt to upload an image with non-ASCII filename to imgur.com using the method from this answer:

Export["Тест.png", Plot[Sin[x], {x, 0, Pi}]]

URLRead[HTTPRequest[
  "http://stackoverflow.com/upload/image", <|
   "Body" -> {"image" -> <|"Content" -> File[%], "MIMEType" -> "image/png"|>}|>]]

screenshot

And undoubtedly there are other components suffering from this bug because reports about problems with Unicode filepaths keep appearing on this site.

Hence it is worth to have a dedicated thread with a collection of general techniques allowing to workaround such problems. This thread is intended exactly for this purpose. Some guidelines:

  • When posting an OS-specific workaround, please include information about OS.
  • If a workaround is limited to local file paths and doesn't work for network paths, please mention this.
  • Each answer should contain elaborated description of only one general method along with its limitations.

Related questions:

$\endgroup$
  • 1
    $\begingroup$ On OSX (MMA 11.1): Importing PDF with some Unicode names (e.g. Import["~/Тест.pdf"]) gives the warning Import::general: Could not find the start of the cross reference table and returns an empty list {}. Renaming file to test.pdf fixes the problem. $\endgroup$ – Shadowray Apr 19 '17 at 14:14
  • $\begingroup$ Reported to the tech support as [CASE:3965891]. $\endgroup$ – Alexey Popkov Oct 30 '17 at 7:53
  • $\begingroup$ I am getting reports about MaTeX not working in some situation where the Windows username, and therefore also $TemporaryDirectory, have non-ASCII characters. The root cause seems to be that RunProcess fails if the current directory has non-ASCII characters in its path. But I cannot reproduce this problem. Can you? E.g. try RunProcess["C:\\windows\\System32\\where.exe", ProcessDirectory -> $TemporaryDirectory] (or substitute a path with non-ASCII chars for $TemporaryDirectory if your username is all-ASCII). $\endgroup$ – Szabolcs Nov 18 '17 at 16:59
  • 2
    $\begingroup$ @Kuba The WRI's support response was: "<...> I have filed a report with the development team and added your contact details to the report. We hope to reach out to you once this issue is resolved in a future release of Mathematica. <...>" $\endgroup$ – Alexey Popkov Nov 18 '17 at 20:46
  • 1
    $\begingroup$ Yeah I'm using Win 7 64bit Chinese edition. Just checked in cmd, seems that "C:\\Users\\PC\\DOCUME~1\\测试.txt" is exactly the correct short filename in Windows Chinese edition i.e. non-ASCII characters can be used in the short filename at least in Windows Chinese edition. (BTW if the filename is changed to e.g. "测试测试测试测试.txt" the output will be "C:\\Users\\PC\\DOCUME~1\\测试测~1.TXT. ) $\endgroup$ – xzczd Jul 1 at 11:02
6
$\begingroup$

Windows-specific: use short paths (short filenames, 8.3 file names)

This method should work with all parts of the Mathematica system.

Limitations

  • short path can be found only for existing file/directory

  • short paths exist only if generation of them isn't disabled by administrator (by default it is enabled)

  • may not work for network paths

  • this method isn't reliable for localized versions of Windows because on them the short paths allowed to contain non-ASCII characters (example).

Solutions

There is an undocumented function FileInformation which allows to obtain the short path (works at least starting from Mathematica 11.3, thanks to GenericAccountName):

Export["Тест.txt", ""]
FileInformation["Тест.txt", "AbsoluteShortFileName"]
"Тест.txt"

"D:\\DOCUME~1\\47BB~1.TXT"

Also starting from version 12.0.0 there is an under-documented (but "official") way to obtain the short path:

Information[File["Тест.txt"], "Properties"] // Shallow
{"ObjectType", "FileName", "DirectoryName", "AbsoluteFileName", "AbsoluteShortFileName", "ShortFileName", "FileBaseName", "FileExtension", "CreationDate", "LastAccessDate", <<33>>}
Information[File["Тест.txt"], "AbsoluteShortFileName"]
"D:\\DOCUME~1\\47BB~1.TXT"

Here is a function which expands a file/directory name and then obtains the corresponding short path using cmd.exe by this method (tested on Windows 7 x64 but should work at least starting from Windows XP):

shortPath[longPath_String] /; FileExistsQ[longPath] := 
 Import["!cmd /c for %A in (\"" <> ExpandFileName[longPath] <> "\") do @echo %~sA", "Text"]

Another solution via NETLink` can be found in this great answer by WReach (see also this answer).

Usage

Create a PDF file with Unicode filepath and find the short path to it:

Export["Тест.pdf", Plot[Sin[x], {x, 0, Pi}]]
shortPath["Тест.pdf"]
"Тест.pdf"
"D:\\DOCUME~1\\FF12~1.PDF"

Import this file:

Import[shortPath["Тест.pdf"]]

output

$\endgroup$
  • 3
    $\begingroup$ In v12 you can get "AbsoluteShortFileName" with FileInformation $\endgroup$ – GenericAccountName Jun 14 at 17:44
  • $\begingroup$ @GenericAccountName Thank you, added to the answer. It also works in version 11.3. $\endgroup$ – Alexey Popkov Jul 1 at 7:58
4
$\begingroup$

Import as "String" first, and then apply ImportString or ...

Currently it is the most elegant workaround, and besides that it is expected to be OS-independent.

Limitations

  • This method is intended exclusively for import-related tasks.

  • If the path defined via $TemporaryDirectory contains non-ASCII characters, ImportString will fail because under the hood it creates a temp file in $TemporaryDirectory and then passes its path to the appropriate importer, hence we return back to the situation with non-ASCII file path (checked with Mathematica 8.0.4 and 11.1.0 on Windows 7 x64). A workaround is to Import from a stream with BinaryFormat -> True (see below).

Solution

It is as simple as described in the head of this answer. You Import your file with Unicode path as "String" (what works!), and then do with the obtained string what you had to do with the original file. For example pass it to ImportString if your purpose was to import the file:

ImportString[Import[<path to PDF file>, "String"], "PDF"]

... or directly include it into HTTPRequest instead of symbolic link to the local file (see the corresponding example in the question):

HTTPRequest["http://stackoverflow.com/upload/image", <|
  "Body" -> {"image" -> <|"Content" -> Import["Тест.png", "String"], 
                          "MIMEType" -> "image/png"|>}|>]

With Mathematica 10 or higher instead of Import one can use ReadString what gives identical result and also is a bit faster:

ImportString[ReadString[<path to PDF file>], "PDF"]

A low-level alternative is to OpenRead the file with option BinaryFormat -> True and then Read it as a whole with the RecordSeparators -> {} option (don't forget to Close it afterwards!):

# &[Read[#, Record, RecordSeparators -> {}], Close[#]] &@
 OpenRead[fileName, BinaryFormat -> True]

... or directly Import from the stream:

# &[Import[#, "PDF"], Close[#]] &@OpenRead[<path to PDF file>, BinaryFormat -> True]

Usage

Import a PDF file with Unicode filepath:

Export["Тест.pdf", Plot[Sin[x], {x, 0, Pi}]]

ImportString[Import["Тест.pdf", "String"], "PDF"]
"Тест.pdf"

output

Upload an image with Unicode filepath to imgur.com:

Export["Тест.png", Plot[Sin[x], {x, 0, Pi}]]

URLRead[HTTPRequest[
  "http://stackoverflow.com/upload/image", <|
   "Body" -> {"image" -> <|"Content" -> Import["Тест.png", "String"], 
       "MIMEType" -> "image/png"|>}|>]]
"Тест.png"

output

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.