# Problem importing URL with Greek characters

I'm considering buying a car. So I thought why not make a web-crawler in Mathematica to pile-up car data? Brilliant idea. Then I found this Greek website, gocar.gr, which just so happens to have all the data I need in a convenient form, with URLs following a very consistent progression:

www.gocar.gr/cars/BRAND/MODEL/EDITION

e.g. www.gocar.gr/cars/OPEL/MOKKA/1.7_CDTi_Edition_/.

The problem is when the model or edition contains Greek letters (but not the brand? Oh wait, there are no Greek car brands), e.g. www.gocar.gr/cars/BMW/ΣΕΙΡΑ_3/, in which case Import["URL", "Data"] fails with a FetchURL::conopen error.

It seems to me that this is some kind of encoding problem (it's consistent with Greek characters appearing in the URL and everything else works). I've seen the -kind of- relevant questions about copying non-Unicode text (this and this), but my problem is staying within Mathematica, not copying something out of it (which, by the way, works fine).

So, to reproduce:

Import["http://www.gocar.gr/cars/BMW","Data"]

works, but

Import["http://www.gocar.gr/cars/BMW/ΣΕΙΡΑ_3","Data"]

doesn't.

And my question is: any ideas?

1. This is a Windows 7 / 64-bit computer; formats and location are set to Greek/Greece, Greek keyboard is installed (duh), display language is set to English, Mathematica version 8.

2. I also tried going directly through the JLink with a Java module I found in some other post (can't find it right now; may credit go where credit is due):

Needs["JLink"]

httpGet[url_String] :=JavaBlock @
Module[{http, get},
http = JavaNew["org.apache.commons.httpclient.HttpClient"];
get = JavaNew["org.apache.commons.httpclient.methods.GetMethod", url];
http @ executeMethod[get]; get @ getResponseBodyAsString[]]

followed by:

ImportString[httpGet[URL - HERE], {"HTML", "Data"}]

No luck.

Thank you for reading my rant.

-
This is probably the post you were referring to... –  rm -rf Jun 12 '13 at 17:48
Well, good question -- +1. But, I don't know what to tell you besides that this was apparently a bug in version 8, since it works correctly in version 9. –  Oleksandr R. Jun 12 '13 at 18:06
What if you percent-encode the URLs with Greek stuff? –  Ｊ. Ｍ. Jun 12 '13 at 18:29
@0x4A4D well, "http%3A%2F%2Fwww.gocar.gr%2Fcars%2FBMW%2F%CE%A3%CE%95%CE%99%CE%A1%CE%91_3%2F" doesn't work (as expected; and it's ugly, too), but percent-encoding just e.g. "ΣΕΙΡΑ_3" in UTF8 (as in "%CE%A3%CE%95%CE%99%CE%A1%CE%91_3") does! Now that's mildly sub-optimal, but it does work. Thank you sir, you are a beautiful -unicode- character. (PS: I kind-of tried this by encoding the whole URL in ISO8859-1, which didn't work, and then I gave up. So, thank you for insisting.) –  kalt Jun 12 '13 at 18:51
@kalt percent encoding is only meant for the path elements of the URL (and/or parameters)--not the protocol, domain name, or path separators. That's why you need to encode only the Greek text: it's not a valid URL otherwise. –  Oleksandr R. Jun 12 '13 at 18:55

The specification RFC1738 : "Uniform Resource Locators" states that:

The characters ";", "/", "?", ":", "@", "=" and "&" are the characters which may be reserved for special meaning within a scheme. No other characters may be reserved within a scheme.

[...] only alphanumerics, the special characters "$-_.+!*'(),", and reserved characters used for their reserved purposes may be used unencoded within a URL. Since the list of reserved/special characters is small, we can write an automated solution (instead of having to manually identify the Greek parts and encode like in kalt's answer) using ExternalServiceEncodeString from here as: encodeURL[str_String] := StringReplace[str, x : Except[Alternatives @@ Characters@";/?:@=&$-_.+!*'()"] :>
ExternalServiceEncodeString[x]]

(alphanumerics are handled correctly by EncodeString). We can now directly encode the URL:

encodeURL["http://www.gocar.gr/cars/BMW/ΣΕΙΡΑ_3"]
(* "http://www.gocar.gr/cars/BMW/%CE%A3%CE%95%CE%99%CE%A1%CE%91_3" *)
-
+1, but this is not correct for Unicode-based domain names. Those should be encoded using Punycode, not percent-encoding. –  Oleksandr R. Jun 13 '13 at 10:21

Just in case somebody else needs it, here is a compiled answer. Thanks go out to 0x4A4D (for the actual solution), Michael Pilat (for the JLink part) and everybody else in here for the swift responses.

Since this is apparently a bug of sorts in Mathematica 8, percent encoding the Greek letters in the URL will have to do.

Reciting Michael Pilat's code snippet: