I can understand, if Mathematica does not provide such functionality. It is running on top of an operating system, which delivers all the functionality to do these things, like socket I/O etc.
I don't see the point to do this inside of Mathematica.
What you can do is this:
a) unix plattform
Run["/path/to/wget", "http://www.nytimes.com"];
This is just running wget with the default settings. wget does have a load of options which you can set to modify its result (for instance, if I want to download a webpage and its requisites (css links to other pages) I regularly use wget -E -H -k -K -p).
b) windows plattform
In case you don't want to download wget for windows...
1) write a powershell script (wget.ps1):
(new-object System.Net.WebClient).
DownloadFile($args[0],'C:\tmp\index.html')
2) @Bobthechemist found out how to run this on Windows platform:
Run["powershell.exe -ExecutionPolicy ByPass -file c:\\tmp\\wget.ps1 \ http://www.nytimes.com"]
Once the webpage is downloaded, you can start to do all the extractions you want to do.
Edit 1:
Since I've read in your comments I know that you're about to think of a pure Java solution, you might consider this:
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URL;
import java.net.URLConnection;
public class WebPageSaver {
// public static void main(String args[]) throws Exception {
// saveWebpage("http://www.nytimes.com", "/path/to/your/home/index.html");
// }
public static void saveWebpage(String site, String target ) throws Exception {
OutputStream out = new FileOutputStream(target);
URL url = new URL(site);
URLConnection conn = url.openConnection();
conn.connect();
InputStream is = conn.getInputStream();
byte[] buffer = new byte[4096];
while(true) {
int numBytes = is.read(buffer);
if(numBytes == -1) {
break;
}
out.write(buffer, 0, numBytes);
}
}
}
And then you can use one of the many approaches to link this into you Mathematica environment.
Like:
1) Java-Reloader by Leonid <-- recommended!
2) Hands down approach
P.S.: i wrote the windows part answer just out of my memory and i don't have anything here to verify that this is working...
Import[]
ing as"HTML"
. $\endgroup$