1
$\begingroup$

I am considering the following link:

https://myimmitracker.com/en/ca/trackers/consolidated-e-apr-tracker-express-entry-permanent-residency-application

It produces the following page with a table in browser.

enter image description here

If I use chrome inspector tools to see the URL request this page makes to ask for data, I see it uses the following:

https://myimmitracker.com/ca/trackers/consolidated-e-apr-tracker-express-entry-permanent-residency-application/cases?start=100&filter=%7B%22state%22%3A%5B%22Active%22%5D%7D&sort=%5B%5B%22xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux%22%2C%22desc%22%5D%2C%5B%22updated%22%2C%22desc%22%5D%5D

The response "JSON" from the chrome tools is as follows (I have truncated the full response since it exceeds the character limit allowed by stackexchange):

{"values":[{"comments_count":"0 comments","username":["Sibi Abraham","case-68891"],"watched":false,"dashboard":"/en/dashboard","c_at":"Mar  4, 2019","updated":"about 24 hours","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb 15, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":18,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"2","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"United Arab Emirates"},{"comments_count":"0 comments","username":["kmb","case-68758"],"watched":false,"dashboard":"/en/dashboard","c_at":"Mar  3, 2019","updated":"3 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb 15, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":18,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"3","xocoz-vubum-bifub-mitak-varab-habog-genyl-ginat-kaxyx":"Feb 15, 2019","xufol-codoz-vaban-rahuk-sobuc-tumyn-kesev-tikus-naxex":"In Progress (1)","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"New Zealand"},{"comments_count":"0 comments","username":["Othmane","case-67203"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  9, 2019","updated":"20 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"Morocco","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"3","xufol-codoz-vaban-rahuk-sobuc-tumyn-kesev-tikus-naxex":"In Progress (1)","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Morocco"},{"comments_count":"0 comments","username":["Aele","case-67548"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb 13, 2019","updated":"20 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"Nigeria","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"1","xubot-hatel-matec-vycid-fycel-gugyp-tesag-ludyd-lexux":"Copy of the biographical data page of a passport or travel document, Digital Photo, Educational Credential Assessment (ECA) report, Language test results, Medical examination confirmation, Police certificates, Proof of funds, Proof of studies, Proof of work experience","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Nigeria"},{"comments_count":"0 comments","username":["Prashanth Gopenathan","case-67499"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb 13, 2019","updated":"20 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"CEC","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Canada"},{"comments_count":"2 comments","username":["Belly_BearS","case-65222"],"watched":false,"dashboard":"/en/dashboard","c_at":"Jan 15, 2019","updated":"22 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"Pakistan","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Pakistan"},{"comments_count":"1 comments","username":["adityaljoshi","case-67128"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  8, 2019","updated":"25 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"4","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"India"},{"comments_count":"0 comments","username":["maestro","case-67155"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  9, 2019","updated":"25 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"Cameroon","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"1","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"United States of America"},{"comments_count":"0 comments","username":["Navraj Singh Dhindsa","case-67135"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  8, 2019","updated":"25 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"CEC","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"1","xufol-codoz-vaban-rahuk-sobuc-tumyn-kesev-tikus-naxex":"Not Applicable","xubot-hatel-matec-vycid-fycel-gugyp-tesag-ludyd-lexux":"Copy of the biographical data page of a passport or travel document, Digital Photo, Medical examination confirmation, Police certificates, Proof of funds, Proof of work experience","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Canada","xiden-fadin-lasof-dulaz-mavuk-fyzal-myhuv-humyc-tyxix":"Feb  8, 2019"},{"comments_count":"0 comments","username":["chemenger","case-67130"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  8, 2019","updated":"25 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"United States of America"}],"cases_count":11534,"total_cases_count":16666}

How can I use Wolfram Language's HTTP Requests & Responses to get the response from the URL request the page makes when loading new data on scrolling? Basically I want to get the above JSON response using the Mathematica's HTTP functionality?

Improve this question
$\endgroup$
1
  • $\begingroup$ Have you tried simply Importing the url? Otherwise, look at URLRead. $\endgroup$
    – Carl Lange
    Mar 6, 2019 at 8:01

1 Answer 1

Reset to default
6
$\begingroup$

It looks like this particular request requires you to set a header or two before it will respond the same way it does in the browser. In particular, it looks like it needs you to set the x-requested-by, Accept, and Referer headers. (I discovered this by simply whack-a-moling the headers to see which will give the correct response).

I initially took the URL and used URLParse in order to get a URL association, to which I then added the Headers.

Import[URLRead[<|"Scheme" -> "https", "User" -> None, 
   "Domain" -> "myimmitracker.com", "Port" -> None, 
   "Path" -> {"", "ca", "trackers", 
     "consolidated-e-apr-tracker-express-entry-permanent-residency-\
application", "cases"}, 
   "Query" -> {"start" -> "100", 
     "filter" -> "{\"state\":[\"Active\"]}", 
     "sort" -> 
      "[[\"xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux\",\"\
desc\"],[\"updated\",\"desc\"]]"}, "Fragment" -> None,
   "Headers" -> <|
     "Accept" -> "application/json;charset=UTF-8",
     "referer" -> 
      "https://myimmitracker.com/en/ca/trackers/consolidated-e-apr-\
tracker-express-entry-permanent-residency-application",
     "x-requested-with" -> "XMLHttpRequest"
     |>|>], "RawJSON"]

This returns an association in the form of the JSON that you requested.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Can you please explain if I build the url using your association, URLBuild[<|...|>] then the resulting url is the same as that from the browser. But now if I do URLRead from the build url, URLRead[URLBuild[<|...|>]] I get a different HTTPResponse. Why can't it understand from the build url, the headers and other things? $\endgroup$
    – user13892
    Mar 6, 2019 at 9:11
  • $\begingroup$ @user13892 The headers won't stick around if you do URLBuild on the full association - you need to pass the Headers association as a second argument to URLRead. $\endgroup$
    – Carl Lange
    Mar 6, 2019 at 9:15
  • $\begingroup$ Also what happens if I set "start" to "0" in the query. Why can't it import as RAWJSON in this case? Clearly body of the resulting HTTPResponse show that is its indeed JSON string, URLRead[<|...|>]["Body"]. $\endgroup$
    – user13892
    Mar 6, 2019 at 9:28
  • $\begingroup$ Is there an error message? $\endgroup$
    – Carl Lange
    Mar 6, 2019 at 10:04
  • $\begingroup$ Yes two error messages from Import: Import::utf8expcontinuation: Input is not a valid UTF8 byte sequence. 116 is expected to be a continuation byte. and Import::jsonhintposandchar: An error occurred near character 't', at line 1:57844 $\endgroup$
    – user13892
    Mar 6, 2019 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.