5
$\begingroup$

Everyday I spend a crazy amount of time clicking though each maze of hyperlinks and forms on this horrendous site: https://vaxfinder.mass.gov/

Is there anyway that Mathematica can help speed up the process of finding out if there are available appointments for people with 2+ co-morbidities?

$\endgroup$
1
  • $\begingroup$ Perhaps you could describe how you go through the site manually, so that we can help you codify? At present your question doesn't have quite enough detail (for me, a non-US citizen) to help you easily. $\endgroup$
    – Carl Lange
    Feb 23, 2021 at 8:05

1 Answer 1

6
$\begingroup$

Disclaimer: This answer and all of the codes created only for educational purposes and the author does not hold any responsibility.

WARNING: Since running the codes below will put heavy pressure on site and may prevent others who are really in need of the vaccine from accessing the site, PLEASE DO NOT RUN IT REGULARLY.

Base on what I see, you check all the places for availability. Mathematica has different ways to extract data from the web, this solution uses StartWebSession which introduced in version 12. basically, it will open a new browser, in this case, Firefox but you can change it to Chrome (step 3) and browse the pages automatically and will create a dataset of all the places:

1- Defined a function to grab links of individual pages:

grabLinks[session_WebSessionObject, url_ : Null] := Module[{},
  If[url != Null, WebExecute[session, "OpenPage" -> url];];
  WebExecute[session, "JavascriptExecute" -> "temp00=[];
document.querySelectorAll('a.btn.secondary').forEach(function(elem,i){
  temp00.push(elem.href);
});return temp00;"]]

2- Another function to extract table from each page:

extractInformation[session_WebSessionObject, url_String] := 
 Module[{name, response, t1},
  WebExecute[session, "OpenPage" -> url];
  name = First@WebExecute[session, "ElementText" -> "Tag" -> "h1"];
  response = 
   WebExecute[session, 
    "JavascriptExecute" -> 
     "if(document.querySelectorAll('div.table-empty').length==1){
  return 'Empty';
}else{return document.querySelector('table').innerText;}"];
  If[response == "Empty", 
   t1 = {<|"Date" -> "-", "Hours" -> "-", "Available" -> "-"|>};,
   t1 = SemanticImportString[response, {"Date", "Text", "Integer"}, 
      "NamedRows"];];
  Return[Prepend[#, "Name" -> name] & /@ t1]]

3- Start a session and open the url:

mainSession = StartWebSession["Firefox"];

WebExecute[mainSession, "OpenPage" -> "https://vaxfinder.mass.gov"];

4- Extract the number of pages and populate page links base on that:

numberOfPages = 
  StringCases[First@WebExecute[mainSession, "ElementText" -> "CSSSelector" -> "p.current"], 
     DigitCharacter ..] // Last // ToExpression;

pages = Table["https://vaxfinder.mass.gov/?page=" <> ToString@i, {i, 2, numberOfPages}];

5- Extract individual links (places) from pages:

links = Flatten[{grabLinks[mainSession], grabLinks[mainSession, #] & /@ pages}];

6- apply extractInformation on all the links and show the results:

result = extractInformation[mainSession, #] & /@ links;

Dataset @ Flatten @ result

7- exit the browser:

DeleteObject[mainSession];

As I explained in the WARNING, I didn't run it in full capacity, here are two sample outputs:

enter image description here

enter image description here

If you want to check a particular pages, just slice pages before step 5 (starts from page 2 to total pages):

(* only pages 1, 2 ,3 *)
pages = pages[[2;;3]];

(* only pages 2,3 *)
pages = pages[[2;;3]];
links = links[[26;;]]; (* do it after step 5 *)

The output is a dataset, so you can easily filter those places which have availability:

Dataset[Flatten@result][Select[#Available > 0 &]]

Full code:

grabLinks[session_WebSessionObject, url_ : Null] := Module[{},
  If[url != Null, WebExecute[session, "OpenPage" -> url];];
  WebExecute[session, "JavascriptExecute" -> "temp00=[];
document.querySelectorAll('a.btn.secondary').forEach(function(elem,i){
  temp00.push(elem.href);
});return temp00;"]]

extractInformation[session_WebSessionObject, url_String] := 
 Module[{name, response, t1},
  WebExecute[session, "OpenPage" -> url];
  name = First@WebExecute[session, "ElementText" -> "Tag" -> "h1"];
  response = 
   WebExecute[session, 
    "JavascriptExecute" -> 
     "if(document.querySelectorAll('div.table-empty').length==1){
  return 'Empty';
}else{return document.querySelector('table').innerText;}"];
  If[response == "Empty", 
   t1 = {<|"Date" -> "-", "Hours" -> "-", "Available" -> "-"|>};,
   t1 = SemanticImportString[response, {"Date", "Text", "Integer"}, 
      "NamedRows"];];
  Return[Prepend[#, "Name" -> name] & /@ t1]]


mainSession = StartWebSession["Firefox"];

WebExecute[mainSession, "OpenPage" -> "https://vaxfinder.mass.gov"];

numberOfPages = 
  StringCases[
     First@WebExecute[mainSession, 
       "ElementText" -> "CSSSelector" -> "p.current"], 
     DigitCharacter ..] // Last // ToExpression;

pages = Table["https://vaxfinder.mass.gov/?page=" <> ToString@i, {i, 2, numberOfPages}];

links = Flatten[{grabLinks[mainSession], grabLinks[mainSession, #] & /@ pages}];

result = extractInformation[mainSession, #] & /@ links;

Dataset@Flatten@result

DeleteObject[mainSession];
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.