Is it possible to grab all the links at once on a webpage?

Question

I'm working with some webscraping using Mathematica and have run into a question: is it possible to grab all the links on a webpage that scrolls down? The page I am trying to scrape is this: "http://store.nike.com/us/en_us/pw/mens-tops-t-shirts/7puZobp?ipp=120".

I have created a scraper using XMLObjects and Cases with the following code:

(*Open XML Tree*)
urlShirtsPage = 
  "http://store.nike.com/us/en_us/pw/mens-tops-t-shirts/7puZobp?ipp=\
120";
getXml[anyUrl_] := (
   current = urlShirtsPage;
   page = URLFetch[urlShirtsPage];
   If[StringLength@page == 0, getXml@current, 
    Return@ImportString[page, "XMLObject"]]);
largeXml = getXml[urlShirtsPage];
(*Grab All Links*)
firstScreenLinks = Cases[largeXml,
   XMLElement[
    "div", {"class" -> "grid-item fullSize", "data-pdpurl" -> ___, 
     "data-column-index" -> __, "data-item-index" -> __}, {___}], 
   Infinity];
allLinks = secondScreenLinks = Cases[#,
     XMLElement[
       "div", {"class" -> "grid-item fullSize", 
        "data-pdpurl" -> links_, "data-column-index" -> __, 
        "data-item-index" -> __}, {___}] -> links, 1] &@
   firstScreenLinks
Length@allLinks

The code above basically scrapes the URLs of Nike Shirts products that are displayed in columns on the webpage. The only problem is, it only returns 120 URLs when there are 855 products (each with its own respective URLs).

Using the Chrome Developer Tools, I can see that when I scroll a certain distance the webpage pulls up another script that displays additional shirts. I locate the URL that pulls up the new script (using Developer Tools), and Import it into Mathematica using:

urls = "http://store.nike.com/html-services/gridwallData?country=US&\
lang_locale=en_US&gridwallPath=mens-tops-t-shirts/7puZobp&pn=2";
Import[urls, "Text"]

The code above indicates that the new URLs being pulled up are not in the first scraper above, which I assume means it was loaded while I was scrolling down.

I am wondering if there is a faster way to pull up all the URLs on the page (loaded and not yet loaded) rather than having to keep scrolling, checking for new scripts to appear, and importing and scraping the new URLs on Mathematica. As there are over 800 links, it would take me a very long time to perform the latter process. Any insight would be greatly appreciated!

Answer 1

If you look at the URL that you found, you'll notice a parameter pn. If you change this parameter you get another set of t-shirts. To get all t-shirts you need to download that URL for all allowed pn. Let's say pn stands for "page number". Now, how many pages are there?

We have to remember that this is part of an API. Another programmer used the response of this very URL to implement the website. He must have known how to download all t-shirts, so how did he do it? We must investigate.

We download pn=1 and convert the list of rules to an Association object, which is easier to deal with:

startURL = "http://store.nike.com/html-services/gridwallData?country=US&lang_locale=en_US&gridwallPath=mens-tops-t-shirts/7puZobp&pn=1";
res = GeneralUtilities`ToAssociations[Import[startURL, "JSON"]];

We look at the keys of the Association to see what kind of data we have downloaded:

Keys[res]

(* Out: {"heading", "analytics", "trackingData", "seoBlurb", \
"productsEnabled", "cqIncludes", "nextPageDataService", \
"foundResults", "foundProductResults", "sections", "banners", \
"documentTitle", "navigation", "searchReport"} *)

nextPageDataService sounds interesting, let's have a look:

res["nextPageDataService"]

(* Out: "http://store.nike.com/html-services/gridwallData?country=US&lang_\
locale=en_US&gridwallPath=mens-tops-t-shirts/7puZobp&pn=2" *)

It turns out that nextPageDataService gives us the next page's URL.

What we now have to do is download nextPageDataService recursively until the response lacks the nextPageDataService key, which means that we're at the last page. It can be implemented like this:

getPage[url_] := GeneralUtilities`ToAssociations[Import[url, "JSON"]];

data = NestWhileList[
   getPage[#["nextPageDataService"]] &,
   getPage[startURL],
   KeyExistsQ[#, "nextPageDataService"] &
   ];

data now contains all the information available about all t-shirts.

I started by asking what the possible values of pn are. It turns out that we didn't need to know, because the API was recursive by nature. But we can now find it out anyway:

data // Length
(* Out: 15 *)

Which means that pn ranges from 1 to 15.