3
$\begingroup$

I'm having difficulty with Importing urls embedded in a site (with and without pagination). This command:

url = "http://www.worldwideinterweb.com/item/3678-50-horrible-photos-\
    by-horrible-professional-photographers.html";

Length@Union@Import[url, "ImageLinks"]
(*=> Returns 1*)

gives a single url even though there are tons on the site. Moreover, if I manually copy the url of an image, it is not to be found in the set of hyperlinks:

copiedLink = "http://www.worldwideinterweb.com/images/blogphotos/Funny/Professional%20Photography%20Nightmares/awkward%20pregnany%20photos.png";

MemberQ[Import[url, "Hyperlinks"], copiedLink]
(*=> Returns False*)

I guess I'm probably using it wrong and I'm not sure what the difference is between FetchURL[url] and Import[url, "Text"].

$\endgroup$

3 Answers 3

5
$\begingroup$

The link to the picture itself is stored in a data-src element of each hyperlink. You will have to use string patterns to extract them. For instance, you could use the following as a starting point:

url = "http://www.worldwideinterweb.com/item/3678-50-horrible-photos-by-horrible-professional-photographers.html";

StringCases[
  Import[url, "Source"], 
  "data-src=\"/images/blogphotos/Funny/Professional%20Photography%20Nightmares/" 
     ~~ Shortest[link__] ~~ "\"" -> link,
  Infinity
]

{"worst%20redneck%20photography.png", "you%20are%20not%20a%20photographer%20photos.png", "tiger%20photography.png", ...

You can also use URLDecode to remove the "%20" code if you so desire, and StringJoin to reform a full URL from the filenames.

$\endgroup$
3
$\begingroup$

As others have explained, the basic web page that gets imported doesn't contain any images. This is because many (most) web pages are not served as static entities, but have dynamic behavior due to javascript execution. So Import is not going to be terribly useful here.

However, when you use my WebUnit package, this is easy: WebUnit talks directly to a rendered web page and can extract any element from it and even modify dynamic pages.

Download and install as described in the linked post. Then evaluate, in four input cells:

Needs["WebUnit`"];

InstallWebUnit[];

StartWebSession[];

OpenWebPage["http://www.worldwideinterweb.com/item/3678-50-horrible-photos-%20%20%20%20by-horrible-professional-photographers.html"];

And then:

res = JavascriptExecute["
 var imgList = document.getElementsByTagName('img');
 var imgSrc = [];
 for(var i=0; i<imgList.length; i++) {
  imgSrc[i] = imgList[i].src;
 }
 return imgSrc;
"]

(* gives: a list of 119 image links *)

And then you can import the images:

Import /@ RandomSample[Union[res], 3]

(* Returns three images which are not very appropriate to reproduce here *)
$\endgroup$
1
$\begingroup$

Looking at the result of

Union@Import[url, "ImageLinks"]

appears to give an answer:

{"http://www.worldwideinterweb.com/plugins/system/lazyloadforjoomla/blank.gif"}

It is very likely the images are loaded on demand by the scripts on the page. So,

Import[..., "ImageLinks"]

can't be used to spider through them.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.