13
$\begingroup$

How would I get a list of filenames available at a URL? For example, list the files at:

http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/

$\endgroup$
1
  • $\begingroup$ i know javascript : document.getElementsByTagName('a').length - 4 removed 4 because of parent directory ,and 4 label links $\endgroup$ Oct 2, 2013 at 18:04

4 Answers 4

19
$\begingroup$

Does this do the job?

links = Import[
    "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/",
    "Hyperlinks"]

 {"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/?C=N;O=D",
  "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/?C=M;O=A",
  "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/?C=S;O=A",
  ...

then you can cherry-pick the ones you want using, eg,StringCases:

DeleteCases[StringCases[links, __ ~~ "gif"], {}]

{{"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050416-pg.gif"}, 
 {"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050423-pg.gif"}, 
 {"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050430-pg.gif"}, 
 {"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050507-pg.gif"}, ...
$\endgroup$
3
  • $\begingroup$ Much easier than mine :) I am sure that I looked through the docs of Import and couldn't find a reference to "Hyperlinks". If I had, I wouldn't have gone down the R route. $\endgroup$ Oct 2, 2013 at 10:04
  • $\begingroup$ It does return some links that aren't files though such as "www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/…" $\endgroup$ Oct 2, 2013 at 10:14
  • $\begingroup$ @WalkingRandomly True, but that's easily fixed. $\endgroup$
    – cormullion
    Oct 2, 2013 at 10:51
7
$\begingroup$

As is commonly recommended in other Q&As for tasks such as this, we can also use Cases:

baseurl = "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/";
source = Import[baseurl, "XMLObject"];
baseurl <> # & /@ Rest@Cases[source, XMLElement["td", __,
     {XMLElement["a", __, {___ ..., c_, ___ ...}]}
     ] :> c, Infinity]

{"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050101.wkly", "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050108.wkly", "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050115.wkly", "http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/pdi20050122.wkly",...

Or just remove baseurl <> # & /@ if you don't want the full URLs, but only the file names.

$\endgroup$
6
$\begingroup$

Although I am sure that this can be done in Mathematica, I already know how to do it in R. For example, see the answer at https://stackoverflow.com/questions/12930933/retrieve-the-list-of-files-from-a-url

Thanks to RLink, I can use this solution directly in Mathematica

(*Initialise R*)
Needs["RLink`"]
InstallR[]
(*Install the XML package*)
REvaluate["install.packages('XML')"]
rdata = REvaluate["
   {
   library(XML)
   readHTMLTable(\"http://www1.ncdc.noaa.gov/pub/data/cmb/drought/\
weekly-palmers/2005/\", 
                 skip.rows=1:2)[[1]]$Name -> file.list
   }
   "];
filenames = DeleteCases[RFactorToVector[rdata], _Missing];

Lets look at the first 10 filenames:

In[98]:= filenames[[1 ;; 10]]

Out[98]= {"pdi20050101.wkly", "pdi20050108.wkly", "pdi20050115.wkly", \
"pdi20050122.wkly", "pdi20050129.wkly", "pdi20050205.wkly", \
"pdi20050212.wkly", "pdi20050219.wkly", "pdi20050222.wkly", \
"pdi20050226.wkly"}
$\endgroup$
7
  • 1
    $\begingroup$ +1. I do anticipate though more flames from Mac users, for the install.packages part, and they would be right ... $\endgroup$ Oct 2, 2013 at 10:00
  • $\begingroup$ Completely! Is the Mac version of Mathematica any cheaper? I guess it should be if there is less functionality $\endgroup$ Oct 2, 2013 at 10:15
  • $\begingroup$ This was just very unfortunate - this was probably the single most important flaw that I didn't have the time to fix. I do hope to have the time to address this for the next version, though. $\endgroup$ Oct 2, 2013 at 10:25
  • 3
    $\begingroup$ Re:RLink - thanks! Particularly great to hear it from you, given your very diverse background and interests in computational math tools and languages. There are tons of things it currently lacks, of course, but it has been a cool project. Re: PythonLink - not currently, alas, but possibly in the future. My own thinking is that the more links we have the better. That would allow one to exploit Mathematica as a hub for multi-language development, the role which IMO would suit it very well, and a direction which has been so far largely unexplored, particularly in the industry sector. $\endgroup$ Oct 2, 2013 at 10:38
  • 2
    $\begingroup$ 'fully working code in another language' - yep, totally agree. Mathematica can be a nice high-level interface to test and glue code written in a number of different languages, and add extra level of interactivity. $\endgroup$ Oct 2, 2013 at 10:52
0
$\begingroup$

I like this way. It's similar to @Anon but with simpler Cases syntax:

baseurl="http://www1.ncdc.noaa.gov/pub/data/cmb/drought/weekly-palmers/2005/";
source=Import[baseurl,"XMLObject"];
files=Cases[source,{_,"href"->file_/;StringMatchQ[file,"*.*"]}:> file ,\[Infinity]];
baseurl<>#&/@files

And you could easily choose file format, using for example "*.gif" instead of "*.*"

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.