4
$\begingroup$

Given this list of cities:

cities = {"Al-Safira", "Al-Nabk", "Qatana", "Al-Qutayfah", "Al-Tall", 
   "Yabrud", "Zabadani", "Izra", "Al-Sanamayn", "Hama", "Masyaf", 
   "Mudardeh", "Salamiyah", "Al-Suqalaybiyah", "Homs", "Al-Mukharram",
    "Palmyra", "Al-Qusayr", "Talkalakh", "Idlib", "Arihah", 
   "Jisr al-Shugur", "Latakia", "Al-Haffah", "Jableh", "Qardaha", 
   "Quneitra", "As-Suwayda", "Salkhad", "Shahba", "Tartus", "Baniyas",
    "Duraykish", "Safita", "Al-Shaykh Badr", "Atarib", 
   "South Mount Simeon", "Abu Kamal", "Mayadin", "Al-Rastan", "Harem",
    "Ma'arrat al-Numan", "Fiq", "Al-Bab", "Dayr Hafir", "Jarabulus", 
   "Manbij", "Al-Raqqah", "Tell Abyad", "Al-Tharwah", "Afrin", 
   "Ayn Al-Arab", "Al-Malikiyah", "Ra's al-'Ayn", 
   "South Mount Simeon", "North Mount Simeon", "Aleppo", "Azaz", 
   "Darayya", "Duma", "Qudsaya", "Markaz Rif Dimashq", "Daraa", 
   "Deir ez-Zor", "Al-Hasakah", "Al Qamishli"};

Can I create a web-crawler to search on bbc.co.uk/news in order to produce a table which shows how many times each city is mentioned on their website?

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ It is within the capabilities of Mathematica to do this, some might suggest that there are other environments in which this task might be easier. This choice would depend on your skill set. $\endgroup$
    – image_doctor
    Commented May 20, 2014 at 8:35
  • $\begingroup$ Could you perhaps suggest the easiest platform to do this? $\endgroup$
    – Jason Born
    Commented May 20, 2014 at 13:39
  • $\begingroup$ @Öskå No, it is not. $\endgroup$
    – Jason Born
    Commented May 20, 2014 at 20:00
  • $\begingroup$ No, it's 'not a must have'. Apologies for the confusion. $\endgroup$
    – Jason Born
    Commented May 20, 2014 at 21:01
  • $\begingroup$ Did you manage it? :/ $\endgroup$
    – Jason Born
    Commented May 21, 2014 at 13:49

1 Answer 1

Reset to default
8
$\begingroup$

Warning: Composed cities/regions like "South Mount Simeon" have to be checked. I assume that it searches for "South", "Mount" as well so it was yielding huge numbers. I changed them to "South+Mount+Simeon"

todaysDate = {"2014", "05", "21"};
fromDate = {"2011", "01", "01"};
cities = {"Al-Safira", "Al-Nabk", "Qatana", "Al-Qutayfah", "Al-Tall", 
   "Yabrud", "Zabadani", "Izra", "Al-Sanamayn", "Hama", "Masyaf", 
   "Mudardeh", "Salamiyah", "Al-Suqalaybiyah", "Homs", "Al-Mukharram",
    "Palmyra", "Al-Qusayr", "Talkalakh", "Idlib", "Arihah", 
   "Jisr+al-Shugur", "Latakia", "Al-Haffah", "Jableh", "Qardaha", 
   "Quneitra", "As-Suwayda", "Salkhad", "Shahba", "Tartus", "Baniyas",
    "Duraykish", "Safita", "Al-Shaykh+Badr", "Atarib", 
   "South+Mount+Simeon", "Abu+Kamal", "Mayadin", "Al-Rastan", "Harem",
    "Ma'arrat+al-Numan", "Fiq", "Al-Bab", "Dayr+Hafir", "Jarabulus", 
   "Manbij", "Al-Raqqah", "Tell+Abyad", "Al-Tharwah", "Afrin", 
   "Ayn+Al-Arab", "Al-Malikiyah", "Ra's+al-'Ayn", 
   "South+Mount+Simeon", "North+Mount+Simeon", "Aleppo", "Azaz", 
   "Darayya", "Duma", "Qudsaya", "Markaz+Rif+Dimashq", "Daraa", 
   "Deir+ez-Zor", "Al-Hasakah", "Al-Qamishli"};

After checking www.bbc.co.uk/news one can find that the following syntax leads to a search query:

raw[city_] := 
  Import["http://www.bbc.co.uk/search/news/?q=" <> city <> 
  "&sort=date&start_day=" <> fromDate[[3]] <> "&start_month=" <> 
  fromDate[[2]] <> "&start_year=" <> fromDate[[1]] <> "&end_day=" <> 
  todaysDate[[3]] <> "&end_month=" <> todaysDate[[2]] <> 
  "&end_year=" <> todaysDate[[1]], "Text"]

Digging a little into raw@cities[[1]] shows:

News<span> (5)</span></a>

Thus the following works:

rawCities = raw /@ cities;
occurences[feed_] :=
  ToExpression@
    StringReplace[
     First@StringSplit[
       Extract[StringSplit@feed, 
         First@First@Position[StringPosition[StringSplit@feed, "News<span>"], 
       List[List[__]]] + 1], ")"], "(" | "," -> ""]
occurences[rawCities[[15]]]

1138

occurences /@ rawCities
Thread[{cities, occurences /@ rawCities}] // Short

{5, 0, 6, 0, 0, 1, 48, 1, 1, 450, 0, 0, 1, 0, 1138, 0, 11, 1, 1, 359,
0, 0, 202, 0, 3, 3, 10, 0, 0, 0, 39, 72, 0, 1, 1, 2, 2, 14, 4, 5, 21,
0, 0, 14, 0, 0, 1, 0, 3, 0, 4, 0, 0, 0, 2, 2, 827, 25, 27, 145, 6, 0,
30, 6, 1, 0}

{{Al-Safira, 5},{Al-Nabk, 0},<<62>>,{Al-Hasakah, 1},{Al-Qamishli, 0}}

Following the same idea, one could wonder how many times "string" appears on mathematica.stackexchange.com¹

wordQ = "string";
mmaQueryString = 
  Import["https://mathematica.stackexchange.com/search?q=" <> wordQ, "Text"];

At some point mmaQueryString shows:

    <h2>
        1,692 <span class="results-label">results</span>
    </h2>

Then the following works:

Extract[StringSplit@mmaQueryString, 
  First@First@
    Position[StringPosition[StringSplit[mmaQueryString], "results-label"], 
     List[List[__]]] - 2]

"1,693"

¹ It only works for words that don't belong to

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.