3
$\begingroup$

Given this list of cities:

cities = {"Al-Safira", "Al-Nabk", "Qatana", "Al-Qutayfah", "Al-Tall", 
   "Yabrud", "Zabadani", "Izra", "Al-Sanamayn", "Hama", "Masyaf", 
   "Mudardeh", "Salamiyah", "Al-Suqalaybiyah", "Homs", "Al-Mukharram",
    "Palmyra", "Al-Qusayr", "Talkalakh", "Idlib", "Arihah", 
   "Jisr al-Shugur", "Latakia", "Al-Haffah", "Jableh", "Qardaha", 
   "Quneitra", "As-Suwayda", "Salkhad", "Shahba", "Tartus", "Baniyas",
    "Duraykish", "Safita", "Al-Shaykh Badr", "Atarib", 
   "South Mount Simeon", "Abu Kamal", "Mayadin", "Al-Rastan", "Harem",
    "Ma'arrat al-Numan", "Fiq", "Al-Bab", "Dayr Hafir", "Jarabulus", 
   "Manbij", "Al-Raqqah", "Tell Abyad", "Al-Tharwah", "Afrin", 
   "Ayn Al-Arab", "Al-Malikiyah", "Ra's al-'Ayn", 
   "South Mount Simeon", "North Mount Simeon", "Aleppo", "Azaz", 
   "Darayya", "Duma", "Qudsaya", "Markaz Rif Dimashq", "Daraa", 
   "Deir ez-Zor", "Al-Hasakah", "Al Qamishli"};

Can I create a web-crawler to search on bbc.co.uk/news in order to produce a table which shows how many times each city is mentioned on their website?

$\endgroup$
8
  • 1
    $\begingroup$ It is within the capabilities of Mathematica to do this, some might suggest that there are other environments in which this task might be easier. This choice would depend on your skill set. $\endgroup$ May 20, 2014 at 8:35
  • $\begingroup$ Could you perhaps suggest the easiest platform to do this? $\endgroup$
    – Jason Born
    May 20, 2014 at 13:39
  • $\begingroup$ @Öskå No, it is not. $\endgroup$
    – Jason Born
    May 20, 2014 at 20:00
  • $\begingroup$ No, it's 'not a must have'. Apologies for the confusion. $\endgroup$
    – Jason Born
    May 20, 2014 at 21:01
  • $\begingroup$ Did you manage it? :/ $\endgroup$
    – Jason Born
    May 21, 2014 at 13:49

1 Answer 1

7
$\begingroup$

Warning: Composed cities/regions like "South Mount Simeon" have to be checked. I assume that it searches for "South", "Mount" as well so it was yielding huge numbers. I changed them to "South+Mount+Simeon"

todaysDate = {"2014", "05", "21"};
fromDate = {"2011", "01", "01"};
cities = {"Al-Safira", "Al-Nabk", "Qatana", "Al-Qutayfah", "Al-Tall", 
   "Yabrud", "Zabadani", "Izra", "Al-Sanamayn", "Hama", "Masyaf", 
   "Mudardeh", "Salamiyah", "Al-Suqalaybiyah", "Homs", "Al-Mukharram",
    "Palmyra", "Al-Qusayr", "Talkalakh", "Idlib", "Arihah", 
   "Jisr+al-Shugur", "Latakia", "Al-Haffah", "Jableh", "Qardaha", 
   "Quneitra", "As-Suwayda", "Salkhad", "Shahba", "Tartus", "Baniyas",
    "Duraykish", "Safita", "Al-Shaykh+Badr", "Atarib", 
   "South+Mount+Simeon", "Abu+Kamal", "Mayadin", "Al-Rastan", "Harem",
    "Ma'arrat+al-Numan", "Fiq", "Al-Bab", "Dayr+Hafir", "Jarabulus", 
   "Manbij", "Al-Raqqah", "Tell+Abyad", "Al-Tharwah", "Afrin", 
   "Ayn+Al-Arab", "Al-Malikiyah", "Ra's+al-'Ayn", 
   "South+Mount+Simeon", "North+Mount+Simeon", "Aleppo", "Azaz", 
   "Darayya", "Duma", "Qudsaya", "Markaz+Rif+Dimashq", "Daraa", 
   "Deir+ez-Zor", "Al-Hasakah", "Al-Qamishli"};

After checking www.bbc.co.uk/news one can find that the following syntax leads to a search query:

raw[city_] := 
  Import["http://www.bbc.co.uk/search/news/?q=" <> city <> 
  "&sort=date&start_day=" <> fromDate[[3]] <> "&start_month=" <> 
  fromDate[[2]] <> "&start_year=" <> fromDate[[1]] <> "&end_day=" <> 
  todaysDate[[3]] <> "&end_month=" <> todaysDate[[2]] <> 
  "&end_year=" <> todaysDate[[1]], "Text"]

Digging a little into raw@cities[[1]] shows:

News<span> (5)</span></a>

Thus the following works:

rawCities = raw /@ cities;
occurences[feed_] :=
  ToExpression@
    StringReplace[
     First@StringSplit[
       Extract[StringSplit@feed, 
         First@First@Position[StringPosition[StringSplit@feed, "News<span>"], 
       List[List[__]]] + 1], ")"], "(" | "," -> ""]
occurences[rawCities[[15]]]
1138
occurences /@ rawCities
Thread[{cities, occurences /@ rawCities}] // Short
{5, 0, 6, 0, 0, 1, 48, 1, 1, 450, 0, 0, 1, 0, 1138, 0, 11, 1, 1, 359,
0, 0, 202, 0, 3, 3, 10, 0, 0, 0, 39, 72, 0, 1, 1, 2, 2, 14, 4, 5, 21,
0, 0, 14, 0, 0, 1, 0, 3, 0, 4, 0, 0, 0, 2, 2, 827, 25, 27, 145, 6, 0,
30, 6, 1, 0}

{{Al-Safira, 5},{Al-Nabk, 0},<<62>>,{Al-Hasakah, 1},{Al-Qamishli, 0}}

Following the same idea, one could wonder how many times "string" appears on mathematica.stackexchange.com¹

wordQ = "string";
mmaQueryString = 
  Import["https://mathematica.stackexchange.com/search?q=" <> wordQ, "Text"];

At some point mmaQueryString shows:

    <h2>
        1,692 <span class="results-label">results</span>
    </h2>

Then the following works:

Extract[StringSplit@mmaQueryString, 
  First@First@
    Position[StringPosition[StringSplit[mmaQueryString], "results-label"], 
     List[List[__]]] - 2]
"1,693"

¹ It only works for words that don't belong to

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.