How would I get a list of filenames available at a URL? For example, list the files at:


  • $\begingroup$ i know javascript : document.getElementsByTagName('a').length - 4 removed 4 because of parent directory ,and 4 label links $\endgroup$ – Vishal Sharma Oct 2 '13 at 18:04

Does this do the job?

links = Import[


then you can cherry-pick the ones you want using, eg,StringCases:

DeleteCases[StringCases[links, __ ~~ "gif"], {}]

 {"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050507-pg.gif"}, ...
| improve this answer | |
  • $\begingroup$ Much easier than mine :) I am sure that I looked through the docs of Import and couldn't find a reference to "Hyperlinks". If I had, I wouldn't have gone down the R route. $\endgroup$ – WalkingRandomly Oct 2 '13 at 10:04
  • $\begingroup$ It does return some links that aren't files though such as "www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/…" $\endgroup$ – WalkingRandomly Oct 2 '13 at 10:14
  • $\begingroup$ @WalkingRandomly True, but that's easily fixed. $\endgroup$ – cormullion Oct 2 '13 at 10:51

As is commonly recommended in other Q&As for tasks such as this, we can also use Cases:

baseurl = "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/";
source = Import[baseurl, "XMLObject"];
baseurl <> # & /@ Rest@Cases[source, XMLElement["td", __,
     {XMLElement["a", __, {___ ..., c_, ___ ...}]}
     ] :> c, Infinity]

{"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050101.wkly", "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050108.wkly", "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050115.wkly", "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050122.wkly",...

Or just remove baseurl <> # & /@ if you don't want the full URLs, but only the file names.

| improve this answer | |

Although I am sure that this can be done in Mathematica, I already know how to do it in R. For example, see the answer at https://stackoverflow.com/questions/12930933/retrieve-the-list-of-files-from-a-url

Thanks to RLink, I can use this solution directly in Mathematica

(*Initialise R*)
(*Install the XML package*)
rdata = REvaluate["
                 skip.rows=1:2)[[1]]$Name -> file.list
filenames = DeleteCases[RFactorToVector[rdata], _Missing];

Lets look at the first 10 filenames:

In[98]:= filenames[[1 ;; 10]]

Out[98]= {"pdi20050101.wkly", "pdi20050108.wkly", "pdi20050115.wkly", \
"pdi20050122.wkly", "pdi20050129.wkly", "pdi20050205.wkly", \
"pdi20050212.wkly", "pdi20050219.wkly", "pdi20050222.wkly", \
| improve this answer | |
  • 1
    $\begingroup$ +1. I do anticipate though more flames from Mac users, for the install.packages part, and they would be right ... $\endgroup$ – Leonid Shifrin Oct 2 '13 at 10:00
  • $\begingroup$ Completely! Is the Mac version of Mathematica any cheaper? I guess it should be if there is less functionality $\endgroup$ – WalkingRandomly Oct 2 '13 at 10:15
  • $\begingroup$ This was just very unfortunate - this was probably the single most important flaw that I didn't have the time to fix. I do hope to have the time to address this for the next version, though. $\endgroup$ – Leonid Shifrin Oct 2 '13 at 10:25
  • 3
    $\begingroup$ Re:RLink - thanks! Particularly great to hear it from you, given your very diverse background and interests in computational math tools and languages. There are tons of things it currently lacks, of course, but it has been a cool project. Re: PythonLink - not currently, alas, but possibly in the future. My own thinking is that the more links we have the better. That would allow one to exploit Mathematica as a hub for multi-language development, the role which IMO would suit it very well, and a direction which has been so far largely unexplored, particularly in the industry sector. $\endgroup$ – Leonid Shifrin Oct 2 '13 at 10:38
  • 2
    $\begingroup$ 'fully working code in another language' - yep, totally agree. Mathematica can be a nice high-level interface to test and glue code written in a number of different languages, and add extra level of interactivity. $\endgroup$ – Leonid Shifrin Oct 2 '13 at 10:52

I like this way. It's similar to @Anon but with simpler Cases syntax:

files=Cases[source,{_,"href"->file_/;StringMatchQ[file,"*.*"]}:> file ,\[Infinity]];

And you could easily choose file format, using for example "*.gif" instead of "*.*"

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.