1
$\begingroup$

I am considering the following link:

https://myimmitracker.com/en/ca/trackers/consolidated-e-apr-tracker-express-entry-permanent-residency-application

It produces the following page with a table in browser.

enter image description here

If I use chrome inspector tools to see the URL request this page makes to ask for data, I see it uses the following:

https://myimmitracker.com/ca/trackers/consolidated-e-apr-tracker-express-entry-permanent-residency-application/cases?start=100&filter=%7B%22state%22%3A%5B%22Active%22%5D%7D&sort=%5B%5B%22xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux%22%2C%22desc%22%5D%2C%5B%22updated%22%2C%22desc%22%5D%5D

The response "JSON" from the chrome tools is as follows (I have truncated the full response since it exceeds the character limit allowed by stackexchange):

{"values":[{"comments_count":"0 comments","username":["Sibi Abraham","case-68891"],"watched":false,"dashboard":"/en/dashboard","c_at":"Mar  4, 2019","updated":"about 24 hours","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb 15, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":18,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"2","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"United Arab Emirates"},{"comments_count":"0 comments","username":["kmb","case-68758"],"watched":false,"dashboard":"/en/dashboard","c_at":"Mar  3, 2019","updated":"3 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb 15, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":18,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"3","xocoz-vubum-bifub-mitak-varab-habog-genyl-ginat-kaxyx":"Feb 15, 2019","xufol-codoz-vaban-rahuk-sobuc-tumyn-kesev-tikus-naxex":"In Progress (1)","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"New Zealand"},{"comments_count":"0 comments","username":["Othmane","case-67203"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  9, 2019","updated":"20 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"Morocco","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"3","xufol-codoz-vaban-rahuk-sobuc-tumyn-kesev-tikus-naxex":"In Progress (1)","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Morocco"},{"comments_count":"0 comments","username":["Aele","case-67548"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb 13, 2019","updated":"20 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"Nigeria","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"1","xubot-hatel-matec-vycid-fycel-gugyp-tesag-ludyd-lexux":"Copy of the biographical data page of a passport or travel document, Digital Photo, Educational Credential Assessment (ECA) report, Language test results, Medical examination confirmation, Police certificates, Proof of funds, Proof of studies, Proof of work experience","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Nigeria"},{"comments_count":"0 comments","username":["Prashanth Gopenathan","case-67499"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb 13, 2019","updated":"20 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"CEC","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Canada"},{"comments_count":"2 comments","username":["Belly_BearS","case-65222"],"watched":false,"dashboard":"/en/dashboard","c_at":"Jan 15, 2019","updated":"22 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"Pakistan","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Pakistan"},{"comments_count":"1 comments","username":["adityaljoshi","case-67128"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  8, 2019","updated":"25 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"4","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"India"},{"comments_count":"0 comments","username":["maestro","case-67155"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  9, 2019","updated":"25 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"Cameroon","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"1","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"United States of America"},{"comments_count":"0 comments","username":["Navraj Singh Dhindsa","case-67135"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  8, 2019","updated":"25 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"CEC","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"1","xufol-codoz-vaban-rahuk-sobuc-tumyn-kesev-tikus-naxex":"Not Applicable","xubot-hatel-matec-vycid-fycel-gugyp-tesag-ludyd-lexux":"Copy of the biographical data page of a passport or travel document, Digital Photo, Medical examination confirmation, Police certificates, Proof of funds, Proof of work experience","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Canada","xiden-fadin-lasof-dulaz-mavuk-fyzal-myhuv-humyc-tyxix":"Feb  8, 2019"},{"comments_count":"0 comments","username":["chemenger","case-67130"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  8, 2019","updated":"25 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"United States of America"}],"cases_count":11534,"total_cases_count":16666}

How can I use Wolfram Language's HTTP Requests & Responses to get the response from the URL request the page makes when loading new data on scrolling? Basically I want to get the above JSON response using the Mathematica's HTTP functionality?

$\endgroup$
  • $\begingroup$ Have you tried simply Importing the url? Otherwise, look at URLRead. $\endgroup$ – Carl Lange Mar 6 at 8:01
6
$\begingroup$

It looks like this particular request requires you to set a header or two before it will respond the same way it does in the browser. In particular, it looks like it needs you to set the x-requested-by, Accept, and Referer headers. (I discovered this by simply whack-a-moling the headers to see which will give the correct response).

I initially took the URL and used URLParse in order to get a URL association, to which I then added the Headers.

Import[URLRead[<|"Scheme" -> "https", "User" -> None, 
   "Domain" -> "myimmitracker.com", "Port" -> None, 
   "Path" -> {"", "ca", "trackers", 
     "consolidated-e-apr-tracker-express-entry-permanent-residency-\
application", "cases"}, 
   "Query" -> {"start" -> "100", 
     "filter" -> "{\"state\":[\"Active\"]}", 
     "sort" -> 
      "[[\"xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux\",\"\
desc\"],[\"updated\",\"desc\"]]"}, "Fragment" -> None,
   "Headers" -> <|
     "Accept" -> "application/json;charset=UTF-8",
     "referer" -> 
      "https://myimmitracker.com/en/ca/trackers/consolidated-e-apr-\
tracker-express-entry-permanent-residency-application",
     "x-requested-with" -> "XMLHttpRequest"
     |>|>], "RawJSON"]

This returns an association in the form of the JSON that you requested.

$\endgroup$
  • $\begingroup$ Can you please explain if I build the url using your association, URLBuild[<|...|>] then the resulting url is the same as that from the browser. But now if I do URLRead from the build url, URLRead[URLBuild[<|...|>]] I get a different HTTPResponse. Why can't it understand from the build url, the headers and other things? $\endgroup$ – user13892 Mar 6 at 9:11
  • $\begingroup$ @user13892 The headers won't stick around if you do URLBuild on the full association - you need to pass the Headers association as a second argument to URLRead. $\endgroup$ – Carl Lange Mar 6 at 9:15
  • $\begingroup$ Also what happens if I set "start" to "0" in the query. Why can't it import as RAWJSON in this case? Clearly body of the resulting HTTPResponse show that is its indeed JSON string, URLRead[<|...|>]["Body"]. $\endgroup$ – user13892 Mar 6 at 9:28
  • $\begingroup$ Is there an error message? $\endgroup$ – Carl Lange Mar 6 at 10:04
  • $\begingroup$ Yes two error messages from Import: Import::utf8expcontinuation: Input is not a valid UTF8 byte sequence. 116 is expected to be a continuation byte. and Import::jsonhintposandchar: An error occurred near character 't', at line 1:57844 $\endgroup$ – user13892 Mar 6 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.