10
$\begingroup$

Robert Raguet-Schofield in the Mathematica Blog introduced a way to import tweets from Twitter into Mathematica.

I attempted to use this method. I started with this line of code:

xml = Import["https://twitter.com/statuses/user_timeline.xml?screen_name=WolframResearch"]

and received this message:

FetchURL::httperr: The request to URL https://twitter.com/statuses/user_timeline.xml?screen_name=WolframResearch was not successful. The server returned the HTTP status code 404 ("Not Found").

Then I tried this

xml = Import["https://twitter.com/WolframResearch"]a

and this time I got some output. But when I continue the procedure to extract the tweets with this line

tweets = Cases[xml, XMLElement["text", _, {s_String}] :> s, ∞]

I get an empty set.

How can I import tweets into Mathematica in an effective way, given I am not a familiar with Apachi?

$\endgroup$
  • 1
    $\begingroup$ I think the Twitter API has changed a lot since that blog post. You'll probably need to look at the API documentation and start over. $\endgroup$ – cormullion Jul 18 '13 at 22:56
  • $\begingroup$ It is true that the Twitter API has changed and the old one is deprecated. They now require the user to be authenticated, which makes it a bit more of a job retrieving tweets. MMA has SocialMediaData which lets you authenticate yourself and then get some data, but not tweets. It would be interesting to know if there was a way of using the MMA authentication to get other things from the API. $\endgroup$ – C. E. Jul 18 '13 at 23:11
  • $\begingroup$ Using the second XML function as above, I could import some tweets but not all. And I know there is a function in R to do so. So it shoul be possible to implement it in Mathematica also. $\endgroup$ – Morry Jul 19 '13 at 8:19
  • $\begingroup$ @Morry Screen scraping can be done elegantly in a lot of languages, but it's harder in Mathematica because there aren't any really good libraries although it is absolutely possible. If you keep working with Cases, eventually you surely find a pattern that can yield all the tweets. APIs are always preferable if one can get them to work though. $\endgroup$ – C. E. Jul 19 '13 at 12:18
3
$\begingroup$

One first should access his developer account which is codded here as

token = HTTPClient`OAuthAuthentication[
   "ConsumerKey" -> "your_api_key", 
   "ConsumerSecret" -> "your_api_secret", 
   "RequestEndpoint" -> "https://api.twitter.com/oauth/request_token",
   "AuthorizeEndpoint" -> "https://api.twitter.com/oauth/authorize", 
   "AccessEndpoint" -> "https://api.twitter.com/oauth/access_token"];
url = "https://api.twitter.com/1.1/statuses/retweets_of_me.json";
URLFetch[url, "OAuthAuthentication" -> token]`

Then simply you should use API functions. As sample:

urlList = "https://api.twitter.com/1.1/lists/list.json?screen_name=BarackObama";

Then you should take the name of the Slug from the output and then

urlTweet = "https://api.twitter.com/1.1/lists/statuses.json?slug=ofa-legacy-conference&owner_screen_name=BarackObama&per_page=20&page=24"
xml = URLFetch[urlTweet, "OAuthAuthentication" -> token];

Then you need a simple StringCases analysis to get the tweets!

$\endgroup$
  • $\begingroup$ You figured it out. Great answer! :) $\endgroup$ – C. E. Jul 25 '13 at 10:54
2
$\begingroup$

Perhaps it should be mentioned that since this question was active, (prior to version 10) Mathematica has introduced ServiceConnect which allows a user to access the twittersphere and handles the authentication issue. The documentation provides an example of importing tweets as the OP requested.

$\endgroup$
  • $\begingroup$ Could you include a valid example, for completeness? $\endgroup$ – Kuba Mar 8 at 12:17
0
$\begingroup$

Here's a quick hack that nearly answers the question:

xml = Import["http://twitter.com/WolframResearch", "XMLObject"];
tweet = Cases[xml, 
  XMLElement[
    "p", {"class" -> "js-tweet-text tweet-text"}, {tweet__, 
     XMLElement["a", details__]}] :> {tweet, details }, Infinity]

resulting in:

{{"Perfect your parking moves with this new Wolfram Demonstration! ", 
 {"shape" -> "rect", "class" -> "twitter-timeline-link", 
   "dir" -> "ltr", "href" -> "http://t.co/YAa2poc8qi", 
   "rel" -> "nofollow", 
   "data-expanded-url" -> "http://wolfr.am/14dZd2G", 
   "target" -> "_blank", 
   "title" -> "http://wolfr.am/14dZd2G"}, {XMLElement[
    "span", {"class" -> "invisible"}, {"http://"}], 
   XMLElement["span", {"class" -> "tco-ellipsis"}, {}], 
   XMLElement[
    "span", {"class" -> "js-display-url"}, {"wolfr.am/14dZd2G"}], 
   XMLElement["span", {"class" -> "invisible"}, {}], 
   XMLElement[
    "span", {"class" -> "tco-ellipsis"}, {XMLElement[
      "span", {"class" -> 
        "invisible"}, {" "}]}]}},
{"In case you missed it, here's our latest blog post on using 
  Mathematica to do random walks of ", 
  XMLElement[

... and so on

Looks like there's hours of 'amusement' there if you like decoding Symbolic XML!

$\endgroup$
  • $\begingroup$ Is it possible to push the code in a way to extend the results to get the whole tweets? I mean what you proposed is just taking the first split of the tweets, not the whole history of tweets!! $\endgroup$ – Morry Jul 19 '13 at 14:27
  • $\begingroup$ I don't think they would do that. Without the Api you may be restricted. Google brings up many non-Mathematica discussions, eg this one $\endgroup$ – cormullion Jul 19 '13 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.