Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

How would I get a list of filenames available at a URL? For example, list the files at:

http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/

share|improve this question
    
i know javascript : document.getElementsByTagName('a').length - 4 removed 4 because of parent directory ,and 4 label links –  vishal sharma Oct 2 '13 at 18:04
add comment

4 Answers 4

up vote 18 down vote accepted

Does this do the job?

links = Import[
    "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/",
    "Hyperlinks"]

 {"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/?C=N;O=D",
  "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/?C=M;O=A",
  "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/?C=S;O=A",
  ...

then you can cherry-pick the ones you want using, eg,StringCases:

DeleteCases[StringCases[links, __ ~~ "gif"], {}]

{{"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050416-pg.gif"}, 
 {"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050423-pg.gif"}, 
 {"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050430-pg.gif"}, 
 {"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050507-pg.gif"}, ...
share|improve this answer
    
Much easier than mine :) I am sure that I looked through the docs of Import and couldn't find a reference to "Hyperlinks". If I had, I wouldn't have gone down the R route. –  WalkingRandomly Oct 2 '13 at 10:04
    
It does return some links that aren't files though such as "www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/…; –  WalkingRandomly Oct 2 '13 at 10:14
    
@WalkingRandomly True, but that's easily fixed. –  cormullion Oct 2 '13 at 10:51
add comment

As is commonly recommended in other Q&As for tasks such as this, we can also use Cases:

baseurl = "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/";
source = Import[baseurl, "XMLObject"];
baseurl <> # & /@ Rest@Cases[source, XMLElement["td", __,
     {XMLElement["a", __, {___ ..., c_, ___ ...}]}
     ] :> c, Infinity]

{"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050101.wkly", "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050108.wkly", "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050115.wkly", "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050122.wkly",...

Or just remove baseurl <> # & /@ if you don't want the full URLs, but only the file names.

share|improve this answer
add comment

Although I am sure that this can be done in Mathematica, I already know how to do it in R. For example, see the answer at http://stackoverflow.com/questions/12930933/retrieve-the-list-of-files-from-a-url

Thanks to RLink, I can use this solution directly in Mathematica

(*Initialise R*)
Needs["RLink`"]
InstallR[]
(*Install the XML package*)
REvaluate["install.packages('XML')"]
rdata = REvaluate["
   {
   library(XML)
   readHTMLTable(\"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/\
weekly-palmers/2005/\", 
                 skip.rows=1:2)[[1]]$Name -> file.list
   }
   "];
filenames = DeleteCases[RFactorToVector[rdata], _Missing];

Lets look at the first 10 filenames:

In[98]:= filenames[[1 ;; 10]]

Out[98]= {"pdi20050101.wkly", "pdi20050108.wkly", "pdi20050115.wkly", \
"pdi20050122.wkly", "pdi20050129.wkly", "pdi20050205.wkly", \
"pdi20050212.wkly", "pdi20050219.wkly", "pdi20050222.wkly", \
"pdi20050226.wkly"}
share|improve this answer
1  
+1. I do anticipate though more flames from Mac users, for the install.packages part, and they would be right ... –  Leonid Shifrin Oct 2 '13 at 10:00
    
Completely! Is the Mac version of Mathematica any cheaper? I guess it should be if there is less functionality –  WalkingRandomly Oct 2 '13 at 10:15
    
This was just very unfortunate - this was probably the single most important flaw that I didn't have the time to fix. I do hope to have the time to address this for the next version, though. –  Leonid Shifrin Oct 2 '13 at 10:25
3  
Re:RLink - thanks! Particularly great to hear it from you, given your very diverse background and interests in computational math tools and languages. There are tons of things it currently lacks, of course, but it has been a cool project. Re: PythonLink - not currently, alas, but possibly in the future. My own thinking is that the more links we have the better. That would allow one to exploit Mathematica as a hub for multi-language development, the role which IMO would suit it very well, and a direction which has been so far largely unexplored, particularly in the industry sector. –  Leonid Shifrin Oct 2 '13 at 10:38
2  
'fully working code in another language' - yep, totally agree. Mathematica can be a nice high-level interface to test and glue code written in a number of different languages, and add extra level of interactivity. –  Leonid Shifrin Oct 2 '13 at 10:52
show 2 more comments

I like this way. It's similar to @Anon but with simpler Cases syntax:

baseurl="http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/";
source=Import[baseurl,"XMLObject"];
files=Cases[source,{_,"href"->file_/;StringMatchQ[file,"*.*"]}:> file ,\[Infinity]];
baseurl<>#&/@files

And you could easily choose file format, using for example "*.gif" instead of "*.*"

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.