Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Given this list of cities:

cities = {"Al-Safira", "Al-Nabk", "Qatana", "Al-Qutayfah", "Al-Tall", 
   "Yabrud", "Zabadani", "Izra", "Al-Sanamayn", "Hama", "Masyaf", 
   "Mudardeh", "Salamiyah", "Al-Suqalaybiyah", "Homs", "Al-Mukharram",
    "Palmyra", "Al-Qusayr", "Talkalakh", "Idlib", "Arihah", 
   "Jisr al-Shugur", "Latakia", "Al-Haffah", "Jableh", "Qardaha", 
   "Quneitra", "As-Suwayda", "Salkhad", "Shahba", "Tartus", "Baniyas",
    "Duraykish", "Safita", "Al-Shaykh Badr", "Atarib", 
   "South Mount Simeon", "Abu Kamal", "Mayadin", "Al-Rastan", "Harem",
    "Ma'arrat al-Numan", "Fiq", "Al-Bab", "Dayr Hafir", "Jarabulus", 
   "Manbij", "Al-Raqqah", "Tell Abyad", "Al-Tharwah", "Afrin", 
   "Ayn Al-Arab", "Al-Malikiyah", "Ra's al-'Ayn", 
   "South Mount Simeon", "North Mount Simeon", "Aleppo", "Azaz", 
   "Darayya", "Duma", "Qudsaya", "Markaz Rif Dimashq", "Daraa", 
   "Deir ez-Zor", "Al-Hasakah", "Al Qamishli"};

Can I create a web-crawler to search on bbc.co.uk/news in order to produce a table which shows how many times each city is mentioned on their website?

share|improve this question
1  
It is within the capabilities of Mathematica to do this, some might suggest that there are other environments in which this task might be easier. This choice would depend on your skill set. –  image_doctor May 20 at 8:35
    
Could you perhaps suggest the easiest platform to do this? –  user3482534 May 20 at 13:39
    
@Öskå No, it is not. –  user3482534 May 20 at 20:00
    
No, it's 'not a must have'. Apologies for the confusion. –  user3482534 May 20 at 21:01
    
Did you manage it? :/ –  user3482534 May 21 at 13:49

1 Answer 1

up vote 6 down vote accepted

Warning: Composed cities/regions like "South Mount Simeon" have to be checked. I assume that it searches for "South", "Mount" as well so it was yielding huge numbers. I changed them to "South+Mount+Simeon"

todaysDate = {"2014", "05", "21"};
fromDate = {"2011", "01", "01"};
cities = {"Al-Safira", "Al-Nabk", "Qatana", "Al-Qutayfah", "Al-Tall", 
   "Yabrud", "Zabadani", "Izra", "Al-Sanamayn", "Hama", "Masyaf", 
   "Mudardeh", "Salamiyah", "Al-Suqalaybiyah", "Homs", "Al-Mukharram",
    "Palmyra", "Al-Qusayr", "Talkalakh", "Idlib", "Arihah", 
   "Jisr+al-Shugur", "Latakia", "Al-Haffah", "Jableh", "Qardaha", 
   "Quneitra", "As-Suwayda", "Salkhad", "Shahba", "Tartus", "Baniyas",
    "Duraykish", "Safita", "Al-Shaykh+Badr", "Atarib", 
   "South+Mount+Simeon", "Abu+Kamal", "Mayadin", "Al-Rastan", "Harem",
    "Ma'arrat+al-Numan", "Fiq", "Al-Bab", "Dayr+Hafir", "Jarabulus", 
   "Manbij", "Al-Raqqah", "Tell+Abyad", "Al-Tharwah", "Afrin", 
   "Ayn+Al-Arab", "Al-Malikiyah", "Ra's+al-'Ayn", 
   "South+Mount+Simeon", "North+Mount+Simeon", "Aleppo", "Azaz", 
   "Darayya", "Duma", "Qudsaya", "Markaz+Rif+Dimashq", "Daraa", 
   "Deir+ez-Zor", "Al-Hasakah", "Al-Qamishli"};

After checking www.bbc.co.uk/news one can find that the following syntax leads to a search query:

raw[city_] := 
  Import["http://www.bbc.co.uk/search/news/?q=" <> city <> 
  "&sort=date&start_day=" <> fromDate[[3]] <> "&start_month=" <> 
  fromDate[[2]] <> "&start_year=" <> fromDate[[1]] <> "&end_day=" <> 
  todaysDate[[3]] <> "&end_month=" <> todaysDate[[2]] <> 
  "&end_year=" <> todaysDate[[1]], "Text"]

Digging a little into raw@cities[[1]] shows:

News<span> (5)</span></a>

Thus the following works:

rawCities = raw /@ cities;
occurences[feed_] :=
  ToExpression@
    StringReplace[
     First@StringSplit[
       Extract[StringSplit@feed, 
         First@First@Position[StringPosition[StringSplit@feed, "News<span>"], 
       List[List[__]]] + 1], ")"], "(" | "," -> ""]
occurences[rawCities[[15]]]
1138
occurences /@ rawCities
Thread[{cities, occurences /@ rawCities}] // Short
{5, 0, 6, 0, 0, 1, 48, 1, 1, 450, 0, 0, 1, 0, 1138, 0, 11, 1, 1, 359,
0, 0, 202, 0, 3, 3, 10, 0, 0, 0, 39, 72, 0, 1, 1, 2, 2, 14, 4, 5, 21,
0, 0, 14, 0, 0, 1, 0, 3, 0, 4, 0, 0, 0, 2, 2, 827, 25, 27, 145, 6, 0,
30, 6, 1, 0}

{{Al-Safira, 5},{Al-Nabk, 0},<<62>>,{Al-Hasakah, 1},{Al-Qamishli, 0}}

Following the same idea, one could wonder how many times "string" appears on mathematica.stackexchange.com¹

wordQ = "string";
mmaQueryString = 
  Import["http://mathematica.stackexchange.com/search?q=" <> wordQ, "Text"];

At some point mmaQueryString shows:

    <h2>
        1,692 <span class="results-label">results</span>
    </h2>

Then the following works:

Extract[StringSplit@mmaQueryString, 
  First@First@
    Position[StringPosition[StringSplit[mmaQueryString], "results-label"], 
     List[List[__]]] - 2]
"1,693"

¹ It only works for words that don't belong to

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.