7
$\begingroup$

Every call to WebImageSearch costs money (wolfram service credits), thus, I would like to page through results without having to redownload them.

Given the following search how would one retrieve items 6-10 (without repurchasing the first 5)?

WebImageSearch["desert plains", "Thumbnails", MaxItems -> 5]

enter image description here

Update

Response to @kuba's question in the comments:

enter image description here

Update 2

@b3m2a1's solution doesn't run for me:

vgBlock[fn_, HoldPattern[SetDelayed][lhs_, rhs_], expr_] := Module[
    {$recurse},
	Internal`InheritedBlock[{fn},
    Unprotect[fn];
	(lhs /; ! TrueQ[$recurse] := Block[{$recurse = True}, rhs]);
    DownValues[fn] = SortBy[DownValues[fn], FreeQ[$recurse]];
    expr
   ]
];
vgBlock~SetAttributes~HoldAllComplete;
Options[googleImageSearch2]=Join[Options[WebImageSearch], {"StartingFrom" -> 1}];
googleImageSearch2[params:Except[_?OptionQ].., ops:OptionsPattern[]] :=
 vgBlock[GCSFormatRequestParameters, GCSFormatRequestParameters[e___] := GCSFormatRequestParameters[e] /. ("start" -> "1") -> ("start" -> ToString[OptionValue["StartingFrom"]]),
  WebImageSearch[params,  Sequence @@ FilterRules[{Method -> "Google",  ops}, Options @ WebImageSearch]
  ]
]

When I run googleImageSearch2["sweet dee", "MaxItems" -> 4], the progress bar starts working but then fails and the symbols turn red:

enter image description here

Ah ok, WebImageSearch stopped working because I ran out of service credits:

enter image description here

Probably should have warned me!

$\endgroup$
  • $\begingroup$ Can you add a code which shows $CloudCreditsAvailable changing? I can't reproduce it. And can you confirm that those results are not cached, e.g. when you call MaxItems->10 next time, does it take doubled amount or? $\endgroup$ – Kuba May 11 '18 at 13:31
  • $\begingroup$ @Kuba it's not cloud but service credits - both cost money. $\endgroup$ – M.R. May 11 '18 at 14:10
  • $\begingroup$ Try with less items so not to draw down credits $ServiceCreditsAvailable Put[WebImageSearch["desert plains", "Thumbnails", MaxItems -> 5], LocalObject["myimages"]] $ServiceCreditsAvailable Then Get[LocalObject["myimages"]] $ServiceCreditsAvailable credits remain the same as previous call. $\endgroup$ – Hans May 15 '18 at 2:46
  • $\begingroup$ The only idea I have for getting the next 5 would be to use Except[q]. But the documentation is of no help there the only metadata I could see of use is ByteCount. So Map[ByteCount[#] &, a] where a is the result from Get[...] But I found no good examples. $\endgroup$ – Hans May 15 '18 at 3:27
  • $\begingroup$ The function that implements this (for Google search) is WebSearch`Private`GCSPaginationCalls and it looks like it's terribly designed. The result count is hard coded. The start point can't be set via parameter. Etc. $\endgroup$ – b3m2a1 May 18 '18 at 7:06
9
+50
$\begingroup$

So it looks like the only real way to do this is by hacking the mainframe package that underlies this. Everything seems to be clumsy and hard coded, so I had to dig a little. The following works for Google, but it is easily adaptable to Bing.

The main function of interest is GCSFormatRequestParameters which formats request parameters to Google Custom Search. When you use this it hard codes a "start"->"1". We want to specify our own starting point so we will simply hack this function to do our bidding before it passes the parameters to the API call.

First a function to implement a single Villegas-Gayley redefinition in a general manner:

vgBlock[fn_, 
   HoldPattern[SetDelayed][lhs_, rhs_], 
   expr_
   ] :=
  Module[{$recurse},
   Internal`InheritedBlock[{fn},
    Unprotect[fn];
    (lhs /; ! TrueQ[$recurse] :=
      Block[{$recurse = True}, rhs]);
    DownValues[fn] = SortBy[DownValues[fn], FreeQ[$recurse]];
    expr
    ]
   ];
vgBlock~SetAttributes~HoldAllComplete

Then we'll use this to override the old results:

Options[googleImageSearch2] =
  Join[
   Options[WebImageSearch],
   {
    "StartingFrom" -> 1
    }
   ];
googleImageSearch2[params : Except[_?OptionQ] .., 
  ops : OptionsPattern[]] :=
 vgBlock[GoogleCustomSearchFunctions`GCSFormatRequestParameters,
  GoogleCustomSearchFunctions`GCSFormatRequestParameters[e___] :=

   GoogleCustomSearchFunctions`GCSFormatRequestParameters[e] /. ("start" -> "1") -> ("start" -> 
       ToString[OptionValue["StartingFrom"]]),
  WebImageSearch[params,  
   Sequence @@ 
    FilterRules[{Method -> "Google",  ops}, Options@WebImageSearch]
   ]
  ]

And finally we can test that it works:

googleImageSearch2["doggo", "MaxItems" -> 4]

doggo 4

googleImageSearch2["doggo", "MaxItems" -> 4, "StartingFrom" -> 3]

doggo 4+3

One useful note when digging make sure wrap any calls to WebImageSearch in:

Block[
 {
  WebSearch`Private`GCSPaginationCalls,
  WebSearch`Private`BSPaginationCalls
  },
 doop doop
 ]

So that you can test the parameters without actually searching / downloading the results

$\endgroup$
  • $\begingroup$ This is fairly impressive! $\endgroup$ – Anton Antonov May 18 '18 at 14:28
  • $\begingroup$ I'd like to award you the points, but I can't get this working on my Mathematica OSX 10.13.4 with v11.3 $\endgroup$ – M.R. May 20 '18 at 19:45
  • $\begingroup$ @M.R. which Method to WebImageSearch are you using? It looks like it defaults to Bing. Another possibility is that the definition reordering I attempted to impose isn't working. $\endgroup$ – b3m2a1 May 20 '18 at 19:50
  • $\begingroup$ @b3m2a1 see my update for how it fails, your vgBlock is fancy and deserves the points by itself btw $\endgroup$ – M.R. May 20 '18 at 19:52
  • 1
    $\begingroup$ See my answer to (106068) for another implementation of something similar to your vgBlock. $\endgroup$ – Carl Woll May 20 '18 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.