3
$\begingroup$

I want to fetch the content of a Persian website. I tried

 URLFetch["http://www.tsetmc.com/"]

Why is Mathematica unable to recode the characters in the results retuned? Why do they look strange? The problem is not a matter of the language because, for some Persian websites, the function works. But what goes wrong when I access this particular website?

Here is an extract from the data URLFetch retuned

:\.18ìî?'û \.0fήøoí]Ùim/*¯^ï\é\[CenterDot]º îô \
\.16û\[CenterDot]ÂÞ¥?Þhî\.04í \[RawEscape]kíîfûB¿»Óܣ͸ÿy?¾\
\[Paragraph]ظq©ñ\.b3þ­¥3Ø©¦u5X®këívðoÿ \.06¯ýd-ýD/7køfðó_ü \
\.02¦û+nÜë/.éþ&ênÇÅ zq!
\.16ú½ø \.06[Ù{+íns# Þ½í/ÿ[m~çg\.0b?½\.b4ð\.b3Þðj\.b3ÝÝ»½¸\.b3\
\[PlusMinus]¿\[CenterDot]t¦¿\.06/ðâv\[CenterDot]Ó/Æ_âx\.16û¿íGí¸\.b3\
Improve this question
$\endgroup$

3 Answers 3

Reset to default
11
$\begingroup$

That website delivers the URL with a content-encoding of gzip. Apparently, URLFetch does not automatically handle decompressing this. You can decode it manually:

 tempFile = URLSave["http://www.tsetmc.com/", Close[OpenTemporary[]]];
 data = Import[tempFile, {"GZIP", "Text"}];
 DeleteFile[tempFile]
Improve this answer
$\endgroup$
7
  • $\begingroup$ How did you find out? $\endgroup$
    – Morry
    Jul 26, 2013 at 19:23
  • $\begingroup$ When I save the address below I lose some information of the tables!! http://www.tsetmc.com/Loader.aspxParTree=151311&i=22811176775480091 There should be a way to find all the elements of the page! $\endgroup$
    – Morry
    Jul 26, 2013 at 20:50
  • 1
    $\begingroup$ @Morry I found out by using a handy HTTP sniffer program called Fiddler. This works great with Mathematica if you have your Help/Internet Connectivity/Proxy Settings set for "Use proxy settings from your system or browser". As for losing some info, it's possible that some of what you see when looking at the page in a browser is delivered via JavaScript callbacks after the page loads. Perhaps that site has a more API-friendly way of serving up data. $\endgroup$
    – Todd Gayley
    Jul 26, 2013 at 22:21
  • 1
    $\begingroup$ How may I recognize it? When I inspect the element of the page that I look for in Chrome I find it in Chrome Developer Tools... How should I proceed in an efficient way in Mathematica? $\endgroup$
    – Morry
    Jul 26, 2013 at 22:26
  • $\begingroup$ Could you please indicate a specific element or section of the page that is missing from the Mathematica result, but is present in Chrome? $\endgroup$
    – Todd Gayley
    Jul 27, 2013 at 21:31
7
$\begingroup$

My other answer only addressed the issue of decoding the gzipped content. There is another problem, though, which is that this website uses so-called Dynamic HTML to construct the data that you see in a browser window. In other words, not all the data is delivered in the original HTTP result; rather, it is delivered later by JavaScript callbacks from the browser. What you see in the browser is the assembled product of a program that runs in the browser.

One way to get all the data is to programmatically launch a web browser, tell it to load the page, wait for it to finish all the JavaScript calls needed to acquire all the data, and then ask the browser for the final HTML. This is like automating the "View source" feature in some browsers, and it is often referred to as screen scraping.

On Windows, you can do this easily using .NET/Link:

Needs["NETLink`"];
InstallNET[];
ie = CreateCOMObject["InternetExplorer.Application"];
ie@Navigate["http://www.tsetmc.com/Loader.aspx?ParTree=151311&i=22560050433388046"];
Pause[3];
doc = ie@Document;
html = doc@body@innerHTML;
ReleaseCOMObject[ie]

The html variable now holds the full HTML string.

Note that a Pause is required to ensure that the DHTML callbacks have time to complete. I don't know what a proper pause length would be. A motivated programmer could undoubtedly hook into some sort of "page finished" event that is fired by the browser.

Improve this answer
$\endgroup$
5
  • $\begingroup$ If you mention over the top there are some tabs with further information. Is it possible also to grab the data in the tabs?? $\endgroup$
    – Morry
    Jul 30, 2013 at 10:20
  • $\begingroup$ And to be honest this method even does not include the table that I was talking about!! Since the link is changed, try this link: http://www.tsetmc.com/Loader.aspx?ParTree=151311&i=778253364357513 $\endgroup$
    – Morry
    Jul 30, 2013 at 10:33
  • $\begingroup$ It works for me. Did you see my remark about the Pause length? Try increasing the value. As for getting the tabs, that might be tricky. I played around a bit but didn't have any success. $\endgroup$
    – Todd Gayley
    Jul 30, 2013 at 16:19
  • $\begingroup$ You are right... IT IS WORKING! Do you have any idea how I can get the tabs?? Maybe some hints as I am not much familiar with JAVA!! $\endgroup$
    – Morry
    Jul 30, 2013 at 16:36
  • $\begingroup$ Here is my failed attempt to simulate a click on the DPS tab. This builds on the code in my answer. The click method has no effect: all = doc@all; dpsTabElement = Do[If[all@item[i]@innerText == "DPS", Return[all@item[i]]], {i, 0, all@length - 1}]; dpsTabElement@click[] $\endgroup$
    – Todd Gayley
    Aug 3, 2013 at 5:34
1
$\begingroup$

In 10.3, following should be sufficient to get the desired result:

ImportString[URLFetch["http://www.tsetmc.com/"]]
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.