13
$\begingroup$

I have text that contains a zero-width character that I am not able to remove. A minimum example is in str below where the character is at the start of the text.

str = "​​4-6";
res = ToExpression /@ 
  StringSplit[StringReplace[str, WhitespaceCharacter -> ""], "-"]
{​​4, 6}
Head /@ res
{Symbol, Integer}

This character (or these types of characters) are at various locations in the imported text.

They do not appear to fall into WhitespaceCharacter as StringReplace does not match to them and I don't know their locations before-hand so I can't use StringReplacePart.

How do I clean these unknown zero-width character from strings?

$\endgroup$
  • $\begingroup$ Why not use StringSplit[str, Except[WordCharacter]]? It would remove white space characters including the ones overlooked by WhitespaceCharacters. Your example doesn't look as if you want to remove only the two invisible ones. $\endgroup$ – Jens Jun 23 '17 at 0:35
  • $\begingroup$ @Jens This approach will not work for the actual strings as they contain text as well. It is actually a TSV with these characters peppered throughout the fields for some unknown reason. $\endgroup$ – Edmund Jun 23 '17 at 12:32
  • $\begingroup$ Then maybe you could use the approach in my answer directly upon import, i.e., set the CharacterEncoding -> "ASCII" and then remove the offending \:xxxx. $\endgroup$ – Jens Jun 23 '17 at 13:52
11
$\begingroup$

You can see the zero-width character using FullForm:

str //FullForm

"\:200b\:200b4-6"

So, in this case, you can use the following:

StringReplace[str, "​\:200b"->""] //FullForm

"4-6"

Update

Added a threshold, default 2, for determining "zero" width characters

You can use the following function to look for zero-width characters in your string:

findZeroWidthCharacters[str_, threshold_:2] := With[
    {chars = DeleteDuplicates[Characters[str]]},

    Cases[
        Rasterize[
            Row @ MapThread[
                Tooltip,
                {chars, ToCharacterCode[StringJoin@chars]}
            ],
            "Regions"
        ],
        Rule[{code_, _}, {{l_, _}, {r_, _}}] /; r-l<=threshold :> code
    ]
]

Here I apply it to your string:

findZeroWidthCharacters[str]

{8203}

Here I apply it to a longer string:

s = ExampleData[{"Text", "AliceInWonderland"}] <> FromCharacterCode[{8203, 8204, 8207}];
StringLength[s]

findZeroWidthCharacters[s] //AbsoluteTiming

51725

{0.170908, {8203, 8204, 8207}}

$\endgroup$
  • $\begingroup$ This is too specific to the particular zero-width character in this particular string. I don't know how many other types of these characters are in the string. $\endgroup$ – Edmund Jun 22 '17 at 21:22
  • $\begingroup$ @Kuba The slowest part of the code is the use of Rasterize, so it would be better to only use Rasterize once. $\endgroup$ – Carl Woll Jun 22 '17 at 22:10
  • $\begingroup$ Indeed, rookie mistake, time to sleep then. +1ed $\endgroup$ – Kuba Jun 22 '17 at 22:12
  • $\begingroup$ This is not working with Mma 11.1.0 on Win 8.1 Pro. I get an empty string back. Which version and OS are you using? $\endgroup$ – Edmund Jun 22 '17 at 22:48
  • $\begingroup$ I use M11.1 on OSX. What does Rasterize[ Row[{Tooltip[FromCharacterCode[8203], 8203], Tooltip["a", "a"], Tooltip[FromCharacterCode[8204], 8204]}], "Regions"] return? $\endgroup$ – Carl Woll Jun 22 '17 at 22:59
5
$\begingroup$

how about:

ascii = StringJoin@ FromCharacterCode[Range[0, 127]]; 
 StringReplace[str, c_ /; StringFreeQ[ascii, c] -> ""]
Head /@ ToExpression/@StringSplit[%, "-"]

{Integer, Integer}

or even FromCharacterCode[Select[ToCharacterCode[str], # <= 127 &]]

You might even extend that range if you have printable non-ascii characters. In your example the offending character is # 8203

$\endgroup$
  • $\begingroup$ Interesting approach. Does not seem unicode friendly.. $\endgroup$ – Edmund Jun 23 '17 at 12:37
  • $\begingroup$ right, as I said you may need to adjust the character range to suit your needs. ( You could even turn it around and just filter out 8203 ) $\endgroup$ – george2079 Jun 23 '17 at 13:49
3
$\begingroup$

Following a similar logic to george2079:

StringReplace[ImportString[ExportString[str,"Text", CharacterEncoding->"ASCII"]],
"\\:"~~Repeated[HexadecimalCharacter,4]->""]
$\endgroup$
2
$\begingroup$
clean[x_Symbol] := ToExpression@Last@Characters@ToString@x
clean[x_Integer] := x

Head /@ clean /@ res

{Integer, Integer}

$\endgroup$
  • 1
    $\begingroup$ I do not know the position of these zero-width characters before hand in the string so a position based method such as this will not be effective. Also they occur in text and number strings. I only gave a minimal example to demonstrate the character's existence. $\endgroup$ – Edmund Jun 22 '17 at 21:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.