12
$\begingroup$

I have text that contains a zero-width character that I am not able to remove. A minimum example is in str below where the character is at the start of the text.

str = "​​4-6";
res = ToExpression /@ 
  StringSplit[StringReplace[str, WhitespaceCharacter -> ""], "-"]

{​​4, 6}

Head /@ res

{Symbol, Integer}

This character (or these types of characters) are at various locations in the imported text.

They do not appear to fall into WhitespaceCharacter as StringReplace does not match to them and I don't know their locations before-hand so I can't use StringReplacePart.

How do I clean these unknown zero-width character from strings?

Improve this question
$\endgroup$
3
  • $\begingroup$ Why not use StringSplit[str, Except[WordCharacter]]? It would remove white space characters including the ones overlooked by WhitespaceCharacters. Your example doesn't look as if you want to remove only the two invisible ones. $\endgroup$
    – Jens
    Jun 23, 2017 at 0:35
  • $\begingroup$ @Jens This approach will not work for the actual strings as they contain text as well. It is actually a TSV with these characters peppered throughout the fields for some unknown reason. $\endgroup$
    – Edmund
    Jun 23, 2017 at 12:32
  • $\begingroup$ Then maybe you could use the approach in my answer directly upon import, i.e., set the CharacterEncoding -> "ASCII" and then remove the offending \:xxxx. $\endgroup$
    – Jens
    Jun 23, 2017 at 13:52

4 Answers 4

Reset to default
11
$\begingroup$

You can see the zero-width character using FullForm:

str //FullForm

"\:200b\:200b4-6"

So, in this case, you can use the following:

StringReplace[str, "​\:200b"->""] //FullForm

"4-6"

Update

Added a threshold, default 2, for determining "zero" width characters

You can use the following function to look for zero-width characters in your string:

findZeroWidthCharacters[str_, threshold_:2] := With[
    {chars = DeleteDuplicates[Characters[str]]},

    Cases[
        Rasterize[
            Row @ MapThread[
                Tooltip,
                {chars, ToCharacterCode[StringJoin@chars]}
            ],
            "Regions"
        ],
        Rule[{code_, _}, {{l_, _}, {r_, _}}] /; r-l<=threshold :> code
    ]
]

Here I apply it to your string:

findZeroWidthCharacters[str]

{8203}

Here I apply it to a longer string:

s = ExampleData[{"Text", "AliceInWonderland"}] <> FromCharacterCode[{8203, 8204, 8207}];
StringLength[s]

findZeroWidthCharacters[s] //AbsoluteTiming

51725

{0.170908, {8203, 8204, 8207}}

Improve this answer
$\endgroup$
8
  • $\begingroup$ This is too specific to the particular zero-width character in this particular string. I don't know how many other types of these characters are in the string. $\endgroup$
    – Edmund
    Jun 22, 2017 at 21:22
  • $\begingroup$ @Kuba The slowest part of the code is the use of Rasterize, so it would be better to only use Rasterize once. $\endgroup$
    – Carl Woll
    Jun 22, 2017 at 22:10
  • $\begingroup$ Indeed, rookie mistake, time to sleep then. +1ed $\endgroup$
    – Kuba ♦
    Jun 22, 2017 at 22:12
  • $\begingroup$ This is not working with Mma 11.1.0 on Win 8.1 Pro. I get an empty string back. Which version and OS are you using? $\endgroup$
    – Edmund
    Jun 22, 2017 at 22:48
  • $\begingroup$ I use M11.1 on OSX. What does Rasterize[ Row[{Tooltip[FromCharacterCode[8203], 8203], Tooltip["a", "a"], Tooltip[FromCharacterCode[8204], 8204]}], "Regions"] return? $\endgroup$
    – Carl Woll
    Jun 22, 2017 at 22:59
5
$\begingroup$

how about:

ascii = StringJoin@ FromCharacterCode[Range[0, 127]]; 
 StringReplace[str, c_ /; StringFreeQ[ascii, c] -> ""]
Head /@ ToExpression/@StringSplit[%, "-"]

{Integer, Integer}

or even FromCharacterCode[Select[ToCharacterCode[str], # <= 127 &]]

You might even extend that range if you have printable non-ascii characters. In your example the offending character is # 8203

Improve this answer
$\endgroup$
2
  • $\begingroup$ Interesting approach. Does not seem unicode friendly.. $\endgroup$
    – Edmund
    Jun 23, 2017 at 12:37
  • $\begingroup$ right, as I said you may need to adjust the character range to suit your needs. ( You could even turn it around and just filter out 8203 ) $\endgroup$
    – george2079
    Jun 23, 2017 at 13:49
3
$\begingroup$

Following a similar logic to george2079:

StringReplace[ImportString[ExportString[str,"Text", CharacterEncoding->"ASCII"]],
"\\:"~~Repeated[HexadecimalCharacter,4]->""]
Improve this answer
$\endgroup$
2
$\begingroup$ 
clean[x_Symbol] := ToExpression@Last@Characters@ToString@x
clean[x_Integer] := x

Head /@ clean /@ res

{Integer, Integer}

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I do not know the position of these zero-width characters before hand in the string so a position based method such as this will not be effective. Also they occur in text and number strings. I only gave a minimal example to demonstrate the character's existence. $\endgroup$
    – Edmund
    Jun 22, 2017 at 21:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.