7
$\begingroup$

I'm having trouble with importing HTML from XOWA offline wikipedia database. I set up a local server for datamining, but I can't access it with Mathematica. I can request a HTML page with curl or wget, but Mathematica's Import[] crashes the XOWA server with

failed to process request;
request=<<NULL>>
err_msg=[err 0] <gplx> invalid content_type: line=Content-Type: application/x-www-form-urlencoded request=type: GET url: /en.wikipedia.org/wiki/Test    protocol: HTTP/1.1  host: localhost:8080    user_agent: Wolfram HTTPClient 10.2 accept: */* accept_encoding: <<NULL>>   dnt: false  x_requested_with: <<NULL>>  cookie: <<NULL>>    referer: <<NULL>>   content_length: 0   content_type: <<NULL>>  content_type_boundary: <<NULL>> connection: <<NULL>>    pragma: <<NULL>>    cache_control: <<NULL>>
[trace]:
  gplx.core.net.Http_request_parser.Parse_content_type(Unknown Source)
  gplx.core.net.Http_request_parser.Parse(Unknown Source)
  gplx.xowa.servers.http.Http_server_wkr_v2.Run(Unknown Source)
  gplx.xowa.servers.http.Http_server_wkr_v2.Invk(Unknown Source)
  gplx.core.threads.Thread_adp.run(Unknown Source)
  java.lang.Thread.run(Thread.java:745)

I'm using ImportString[Import["!curl \"http://localhost:8080/en.wikipedia.org/wiki/"<>link<>"\"","Text"],"XMLObject"] at the moment but I'm interested in fixing the problem with direct importing. Unfortunately I don't understand the Java errors so I can't debug this myself.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
10
$\begingroup$

This is an issue with XOWA. The HTTP Server was rewritten in v2.7.2 to handle POSTs and other features. However, it looks like it crashes on your request.

I'll look at fixing this for v2.8.2. I'll comment again here when I have a resolution, but feel free to contact me directly for more info.

Hope this helps!

[Edit: This was fixed for v2.8.2. XOWA now accepts GET requests with a Content-Type. See https://github.com/gnosygnu/xowa/releases ]

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ Thanks for the notice, I look forward to using the fixed server. $\endgroup$
    – shrx
    Aug 3, 2015 at 7:38
  • 1
    $\begingroup$ I have a fix ready for this. I created a ticket here with more info: github.com/gnosygnu/xowa/issues/17 . Basically, XOWA became more strict about the request headers in v2.7.2 and rejected GET requests with headers of Content-Type. curl and get probably work b/c they don't submit Content-Type headers, but Mathetmatica fails. This has been relaxed for v2.8.2 which will be released on Monday. If you want a version before then, let me know what OS and bitness, and I'll make a temporary build for you. $\endgroup$
    – gnosygnu
    Aug 5, 2015 at 3:29
  • 1
    $\begingroup$ Also, are you planning to mine all the <math> statements in English Wikipedia? If so, I can probably put together a sqlite database for you with these columns: page_namespace, page_name, math_source. This will take me 4 days to generate, but it will probably be faster than running it thru the HTTP server. I can also provide instructions as well if that's where you're headed. Let me know if you have any questions. And thanks again for reporting the issue! $\endgroup$
    – gnosygnu
    Aug 5, 2015 at 3:34
  • 1
    $\begingroup$ I am actually interested in creating article link connectivity graphs, inspired by Wikipedia:Getting to Philosophy but also for other common words besides "philosophy". I wrote a pretty robust parser that handles a couple of thousand articles, which I more or less thoroughly checked "by hand", without fail. The only issue is the speed, it can process only 100 - 200 articles per minute. Most of the time is spent requesting the article from the server, processing it to an XMLObject and then parsing it for the correct link.[cont.] $\endgroup$
    – shrx
    Aug 5, 2015 at 6:41
  • 1
    $\begingroup$ [cont.] So I would appreciate if there was a way to create a database containing article's title and its (namespace 0) links (titles, not IDs), however I would still somehow have to check for each link if it satisfies the rules - not italic, not in parentheses, etc. Let me know if it's possible. And also, in the meantime before you push the updated XOWA version, I guess I can just get around this issue if I specify an empty Content-type header? $\endgroup$
    – shrx
    Aug 5, 2015 at 6:44
4
$\begingroup$

As @gnosygnu mentioned in the comments of his answer, the problem is with the "Content-Type" header. So as a temporary fix before XOWA is updated, this works:

ImportString[URLFetch["http://localhost:8080/en.wikipedia.org/wiki/Test",
"Content","Headers"->{"Content-Type"->""}],"XMLObject"]
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.