Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I am trying to download some data from a website that uses "java enabling" to tell me that I cannot download the data. Using

Import[ "website...", "Data"] 

I get

{....."You need Javascript enabled to view live scores. Loading... Loading... ..}

Do I have to use something like squid to change requester_id so it shows up as a web-browser rather than Mathematica ?

I have now fixed this problem. The second problem is that Mathematica does not support iframe. Does it ? Is there a way around this issue ?

share|improve this question
    
Please post your code and the website you are trying to scrape. You should probably post your second question as a separate question. –  Verbeia Mar 12 '12 at 19:55
    
You wrote: "I have now fixed this problem." Please post your solution as an answer to your own question, as soon as the system will allow it (a matter of "reputation" points and time). –  Mr.Wizard Mar 13 '12 at 7:48
    
No, my fault, I have not fixed anything. I deluded myself into believing I had fixed this issue. I just tried a different category on the same site which did not work either but gave the iframe error message. Sorry. –  slyer Mar 13 '12 at 8:17
    
@Verbeia for example, livexscores.com/livescore/football i am into live quotes. –  slyer Mar 13 '12 at 8:29
add comment

1 Answer 1

The user agent (requester_id) probably isn't the issue. Indeed, Mathematica does not appear as a standard browser to web servers (it identifies itself as Mathematica), but that usually isn't an issue for normal HTML pages. And indeed Squid can solve problems if your page is testing for a specific browser (see also an earlier question on user agents)

For pages that are partly generated by a Javascript script this is different, as Mathematica does not interpret Javascript. I don't think you can get this to work.

share|improve this answer
1  
If you are completely crazy, or desperate/curious, you could interface with a javascript interpreter via MathLink, like FireFox's SpiderMonkey. I'm almost tempted to do it, myself, and yes I fall under the crazy/curious header. –  rcollyer Mar 13 '12 at 1:47
    
would there be other ways to scrape the data ? with ruby or scrapy ? –  slyer Mar 13 '12 at 1:47
    
@rcollyer for example something like livexscores.com/livescore/football ? i guess ruby would be the way to go, no ? –  slyer Mar 13 '12 at 8:28
    
nevermind i am closing this question. –  slyer Mar 15 '12 at 17:42
    
I am very curious about updates here, I have mathematica 9 and also cant doload a page because of either iframe or because mathematica identifies as Mathematica not as say Chrome. any updates? –  AimForClarity Jun 2 '13 at 15:00
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.