A quick question, but which I don't believe has been asked here or at SO.

Does Mathematica have a simple way to just download a file from the web? i.e. if I have a list of PDF links (~ 2,000), can I use Mathematica to quickly take them and save them to my system. The obvious way that I've used in the past is to Import[] the data, but since we're talking large-ish PDFs I wonder if there's a way to skip that step.

I have used wget for this sort of thing in the past, but just seeing if there's an easy way to do it within Mathematica. The 'Web Operations' section of the documentation does not seem to have any obvious reference to this. If not, I will obviously just use the 'proper' tool.

-

Needs["UtilitiesURLTools"];
path = FetchURL[
get_obj=ifp2_cut.mesh.gz&acces=ifp2_cut", "ifp2_cut.mesh.gz"];

-
This works great. I've never used this Package, so will read up on it. –  programming_historian Feb 2 '12 at 18:03

As a quick update to this, Mathematica 9 added some additional functionality that I have been putting to use - it seems to work very well if you do text analysis, which I do.

Documentation for the new commands URLSave, URLFetch, and the really useful URLSaveAsynchronous are here. For example, I've been running commands like this:

URLSave["http://activehistory.ca","/users/MYPATH/activehistory.txt"];


Which seem to work perfectly.

-
This is the one I would recommend with V9 and beyond. The FetchURL function above now internally uses URLFetch and similar functions from the new HTTPClient package. It supports proxies and lots of options. –  Joel Klein Jan 16 '13 at 16:49

You can run wget from within Mathematica. For example if you have your URLs in a text file then just run this in a Mathematica input cell (where on my Mac wget is located in sw/bin):

Run["/sw/bin/wget --no-check-certificate --user-agent=" " -x -i /Users/mypath/urls.txt"]


This is actual code -- with the path changed -- that I run to download information from the Australian stock exchange so I can verify that wget can be run from within Mathematica this way. If you are already set up for using wget` then as you probably know it offers advantages over built in Mathematica functions in so far as you can download from sites that require login and do many other useful things.

-
Fantastic! I've used wget before,esp. when I need to use cookies and such, so this is a great addition to my toolkit. Thanks for this. –  programming_historian Feb 2 '12 at 22:33