Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Robert Raguet-Schofield in the Mathematica Blog introduced a way to import tweets from Twitter into Mathematica.

I attempted to use this method. I started with this line of code:

xml = Import["https://twitter.com/statuses/user_timeline.xml?screen_name=WolframResearch"]

and received this message:

FetchURL::httperr: The request to URL https://twitter.com/statuses/user_timeline.xml?screen_name=WolframResearch was not successful. The server returned the HTTP status code 404 ("Not Found").

Then I tried this

xml = Import["https://twitter.com/WolframResearch"]a

and this time I got some output. But when I continue the procedure to extract the tweets with this line

tweets = Cases[xml, XMLElement["text", _, {s_String}] :> s, ∞]

I get an empty set.

How can I import tweets into Mathematica in an effective way, given I am not a familiar with Apachi?

share|improve this question
    
I think the Twitter API has changed a lot since that blog post. You'll probably need to look at the API documentation and start over. –  cormullion Jul 18 '13 at 22:56
    
It is true that the Twitter API has changed and the old one is deprecated. They now require the user to be authenticated, which makes it a bit more of a job retrieving tweets. MMA has SocialMediaData which lets you authenticate yourself and then get some data, but not tweets. It would be interesting to know if there was a way of using the MMA authentication to get other things from the API. –  Pickett Jul 18 '13 at 23:11
    
Using the second XML function as above, I could import some tweets but not all. And I know there is a function in R to do so. So it shoul be possible to implement it in Mathematica also. –  Morry Jul 19 '13 at 8:19
    
@Morry Screen scraping can be done elegantly in a lot of languages, but it's harder in Mathematica because there aren't any really good libraries although it is absolutely possible. If you keep working with Cases, eventually you surely find a pattern that can yield all the tweets. APIs are always preferable if one can get them to work though. –  Pickett Jul 19 '13 at 12:18
add comment

2 Answers

One first should access his developer account which is codded here as

token = HTTPClient`OAuthAuthentication[
   "ConsumerKey" -> "your_api_key", 
   "ConsumerSecret" -> "your_api_secret", 
   "RequestEndpoint" -> "https://api.twitter.com/oauth/request_token",
   "AuthorizeEndpoint" -> "https://api.twitter.com/oauth/authorize", 
   "AccessEndpoint" -> "https://api.twitter.com/oauth/access_token"];
url = "https://api.twitter.com/1.1/statuses/retweets_of_me.json";
URLFetch[url, "OAuthAuthentication" -> token]`

Then simply you should use API functions. As sample:

urlList = "https://api.twitter.com/1.1/lists/list.json?screen_name=BarackObama";

Then you should take the name of the Slug from the output and then

urlTweet = "https://api.twitter.com/1.1/lists/statuses.json?slug=ofa-legacy-conference&owner_screen_name=BarackObama&per_page=20&page=24"
xml = URLFetch[urlTweet, "OAuthAuthentication" -> token];

Then you need a simple StringCases analysis to get the tweets!

share|improve this answer
    
You figured it out. Great answer! :) –  Pickett Jul 25 '13 at 10:54
add comment

Here's a quick hack that nearly answers the question:

xml = Import["http://twitter.com/WolframResearch", "XMLObject"];
tweet = Cases[xml, 
  XMLElement[
    "p", {"class" -> "js-tweet-text tweet-text"}, {tweet__, 
     XMLElement["a", details__]}] :> {tweet, details }, Infinity]

resulting in:

{{"Perfect your parking moves with this new Wolfram Demonstration! ", 
 {"shape" -> "rect", "class" -> "twitter-timeline-link", 
   "dir" -> "ltr", "href" -> "http://t.co/YAa2poc8qi", 
   "rel" -> "nofollow", 
   "data-expanded-url" -> "http://wolfr.am/14dZd2G", 
   "target" -> "_blank", 
   "title" -> "http://wolfr.am/14dZd2G"}, {XMLElement[
    "span", {"class" -> "invisible"}, {"http://"}], 
   XMLElement["span", {"class" -> "tco-ellipsis"}, {}], 
   XMLElement[
    "span", {"class" -> "js-display-url"}, {"wolfr.am/14dZd2G"}], 
   XMLElement["span", {"class" -> "invisible"}, {}], 
   XMLElement[
    "span", {"class" -> "tco-ellipsis"}, {XMLElement[
      "span", {"class" -> 
        "invisible"}, {" "}]}]}},
{"In case you missed it, here's our latest blog post on using 
  Mathematica to do random walks of ", 
  XMLElement[

... and so on

Looks like there's hours of 'amusement' there if you like decoding Symbolic XML!

share|improve this answer
    
Is it possible to push the code in a way to extend the results to get the whole tweets? I mean what you proposed is just taking the first split of the tweets, not the whole history of tweets!! –  Morry Jul 19 '13 at 14:27
    
I don't think they would do that. Without the Api you may be restricted. Google brings up many non-Mathematica discussions, eg this one –  cormullion Jul 19 '13 at 14:39
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.