Import information from a web URL that requires a username and password

Question

I wish to use:

Import[link, "Plaintext"]

to get the plain text on a web page - however the webpage is on an internal network and requires a username and password for access.

I have tried using:

URLFetch[link, "Content", "Username" -> "myusername",  "Password" -> "mypassword"]

and then execute my Import line, but this will not work. Has anyone encountered this problem before?

Answer 1

I managed to figure out what I needed!

Indeed, the answer that worked for me is to pass along the cookie information that contains the login information for the site. Once I pinpointed which cookie was needed (I personally found this information with my Chrome browser in Advanced Settings, under “Privacy and Security” -> "Content Settings” -> “Cookies”), I used Mathematica's SetCookie function. Prior to executing the Import line, I ran something like this:

SetCookies[<|"Domain" -> "[INSERT_VALUE]", "Path" -> "[INSERT_VALUE]", "Name" -> "[INSERT_VALUE]",  "Content" -> "[INSERT_VALUE]"|>];

I replaced all “[INSERT_VALUE]" with the appropriate information about the cookie.

I then executed my

Import[link, "Plaintext"]

line, and voila! I have the information I needed!