5
$\begingroup$

Occasionally when I scrape images from a webpage, such as:

Import["https://www.freeart.com/gallery/m/mondrian/mondrian.html", "Images"]

only a portion of the images are returned (v. 11.3.0.0, MacOS). Is Mathematica imposing some total file size constraint?

One workaround is to download just the internal Hyperlinks on the webpage

Import["https://www.freeart.com/gallery/m/mondrian/mondrian.html", 
"Hyperlinks"]

and then edit (by hand) to pick out the links to relevant images and download those individually, but this seems inelegant.

How can I ensure I scrape all images on a webpage?

Improve this question
$\endgroup$
3
  • $\begingroup$ If you open the "network" tab in your browser (f12 on chrome), then you can see that a lot of the content is not loaded immediately when you go to the page, but rather when you start scrolling down. If you want to get all pages, you would need to imitate scrolling down the page. Potentially with JavaScript, or accessing their API in another way. $\endgroup$
    – enano9314
    Commented Nov 5, 2020 at 20:39
  • $\begingroup$ @enano9314: Thanks. I don't program in JavaScript. Is there some simple modification to the Mathematica Import code to "scroll down"? $\endgroup$
    – David G. Stork
    Commented Nov 5, 2020 at 20:47
  • 1
    $\begingroup$ @DavidG.Stork nope, you need to use WebExecute to do stuff like that $\endgroup$
    – b3m2a1
    Commented Nov 5, 2020 at 21:10

1 Answer 1

Reset to default
4
$\begingroup$

Update Add the image URL's

{https://www.freeart.com/gallery/m/mondrian/mondrian8p.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian1.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian2.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian3.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian4.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian5.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian6.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian7.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian8.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian9.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian12.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian11.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian13.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian17.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian10.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian14.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian15.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian16.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian18.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian74.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian21.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian19.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian20.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian22.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian23.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian24.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian25.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian27.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian32.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian33.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian26.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian28.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian29.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian30.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian31.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian34.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian35.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian36.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian37.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian38.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian39.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian41.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian42.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian43.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian40.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian46.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian44.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian45.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian47.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian48.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian49.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian50.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian51.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian52.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian53.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian76.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian77.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian55.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian75.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian54.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian56.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian65.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian57.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian58.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian59.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian63.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian64.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian62.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian61.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian66.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian67.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian68.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian69.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian70.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian71.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian72.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian60.jpg,https://www.freeart.com/gallery/m/mondrian/smondrian73.jpg}

Using the helper function getWebElementProperty written by @b3m2a1 here.

session = StartWebSession[]
WebExecute[session, "OpenPage" -> "https://www.freeart.com/gallery/m/mondrian/mondrian.html"]

imageTags = WebExecute[session, "LocateElements" -> "Tag" -> "img"];
imageURLs = getWebElementProperty[#, "src"] & /@ imageTags //
  Select[StringContainsQ[#, "https://www.freeart.com/gallery"] &]

gets all of the image URLs which can be used to Import.

Then delete the session

DeleteObject[session]
Improve this answer
$\endgroup$
4
  • $\begingroup$ RohitNamjoshi: Thanks so much. This looks like it may work, but alas I have only v. 11.3.0.0 and StartWebSession was introduced in v. 12. Is there any workaround that will implement your code in v. 11? $\endgroup$
    – David G. Stork
    Commented Nov 5, 2020 at 22:47
  • 1
    $\begingroup$ @DavidG.Stork There is support for this in v11 using StartExternalSession which is documented here. I have no experience using it. There are several related posts on MSE that might help. $\endgroup$
    – Rohit Namjoshi
    Commented Nov 6, 2020 at 3:00
  • $\begingroup$ @DavidG.Stork I added the image URLs in case you needed them. $\endgroup$
    – Rohit Namjoshi
    Commented Nov 7, 2020 at 3:03
  • $\begingroup$ Thanks... but alas I just can't seem to get everything to work. I upvoted your answer, and will accept it once I somehow figure everything out. You've helped a lot, but a solution that includes everything from start to finish that gives me all images on a large webpage would be accepted. $\endgroup$
    – David G. Stork
    Commented Nov 7, 2020 at 4:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.