13
$\begingroup$

How would I get a list of filenames available at a URL? For example, list the files at:

http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/

Improve this question
$\endgroup$
1
  • $\begingroup$ i know javascript : document.getElementsByTagName('a').length - 4 removed 4 because of parent directory ,and 4 label links $\endgroup$
    – Vishal Sharma
    Oct 2, 2013 at 18:04

4 Answers 4

Reset to default
19
$\begingroup$

Does this do the job?

links = Import[
    "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/",
    "Hyperlinks"]

 {"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/?C=N;O=D",
  "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/?C=M;O=A",
  "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/?C=S;O=A",
  ...

then you can cherry-pick the ones you want using, eg,StringCases:

DeleteCases[StringCases[links, __ ~~ "gif"], {}]

{{"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050416-pg.gif"}, 
 {"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050423-pg.gif"}, 
 {"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050430-pg.gif"}, 
 {"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050507-pg.gif"}, ...
Improve this answer
$\endgroup$
3
  • $\begingroup$ Much easier than mine :) I am sure that I looked through the docs of Import and couldn't find a reference to "Hyperlinks". If I had, I wouldn't have gone down the R route. $\endgroup$
    – WalkingRandomly
    Oct 2, 2013 at 10:04
  • $\begingroup$ It does return some links that aren't files though such as "www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/…" $\endgroup$
    – WalkingRandomly
    Oct 2, 2013 at 10:14
  • $\begingroup$ @WalkingRandomly True, but that's easily fixed. $\endgroup$
    – cormullion
    Oct 2, 2013 at 10:51
7
$\begingroup$

As is commonly recommended in other Q&As for tasks such as this, we can also use Cases:

baseurl = "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/";
source = Import[baseurl, "XMLObject"];
baseurl <> # & /@ Rest@Cases[source, XMLElement["td", __,
     {XMLElement["a", __, {___ ..., c_, ___ ...}]}
     ] :> c, Infinity]

{"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050101.wkly", "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050108.wkly", "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050115.wkly", "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050122.wkly",...

Or just remove baseurl <> # & /@ if you don't want the full URLs, but only the file names.

Improve this answer
$\endgroup$
6
$\begingroup$

Although I am sure that this can be done in Mathematica, I already know how to do it in R. For example, see the answer at https://stackoverflow.com/questions/12930933/retrieve-the-list-of-files-from-a-url

Thanks to RLink, I can use this solution directly in Mathematica

(*Initialise R*)
Needs["RLink`"]
InstallR[]
(*Install the XML package*)
REvaluate["install.packages('XML')"]
rdata = REvaluate["
   {
   library(XML)
   readHTMLTable(\"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/\
weekly-palmers/2005/\", 
                 skip.rows=1:2)[[1]]$Name -> file.list
   }
   "];
filenames = DeleteCases[RFactorToVector[rdata], _Missing];

Lets look at the first 10 filenames:

In[98]:= filenames[[1 ;; 10]]

Out[98]= {"pdi20050101.wkly", "pdi20050108.wkly", "pdi20050115.wkly", \
"pdi20050122.wkly", "pdi20050129.wkly", "pdi20050205.wkly", \
"pdi20050212.wkly", "pdi20050219.wkly", "pdi20050222.wkly", \
"pdi20050226.wkly"}
Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ +1. I do anticipate though more flames from Mac users, for the install.packages part, and they would be right ... $\endgroup$
    – Leonid Shifrin
    Oct 2, 2013 at 10:00
  • $\begingroup$ Completely! Is the Mac version of Mathematica any cheaper? I guess it should be if there is less functionality $\endgroup$
    – WalkingRandomly
    Oct 2, 2013 at 10:15
  • $\begingroup$ This was just very unfortunate - this was probably the single most important flaw that I didn't have the time to fix. I do hope to have the time to address this for the next version, though. $\endgroup$
    – Leonid Shifrin
    Oct 2, 2013 at 10:25
  • 3
    $\begingroup$ Re:RLink - thanks! Particularly great to hear it from you, given your very diverse background and interests in computational math tools and languages. There are tons of things it currently lacks, of course, but it has been a cool project. Re: PythonLink - not currently, alas, but possibly in the future. My own thinking is that the more links we have the better. That would allow one to exploit Mathematica as a hub for multi-language development, the role which IMO would suit it very well, and a direction which has been so far largely unexplored, particularly in the industry sector. $\endgroup$
    – Leonid Shifrin
    Oct 2, 2013 at 10:38
  • 2
    $\begingroup$ 'fully working code in another language' - yep, totally agree. Mathematica can be a nice high-level interface to test and glue code written in a number of different languages, and add extra level of interactivity. $\endgroup$
    – Leonid Shifrin
    Oct 2, 2013 at 10:52
0
$\begingroup$

I like this way. It's similar to @Anon but with simpler Cases syntax:

baseurl="http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/";
source=Import[baseurl,"XMLObject"];
files=Cases[source,{_,"href"->file_/;StringMatchQ[file,"*.*"]}:> file ,\[Infinity]];
baseurl<>#&/@files

And you could easily choose file format, using for example "*.gif" instead of "*.*"

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.