10
$\begingroup$

Robert Raguet-Schofield in the Mathematica Blog introduced a way to import tweets from Twitter into Mathematica.

I attempted to use this method. I started with this line of code:

xml = Import["https://twitter.com/statuses/user_timeline.xml?screen_name=WolframResearch"]

and received this message:

FetchURL::httperr: The request to URL https://twitter.com/statuses/user_timeline.xml?screen_name=WolframResearch was not successful. The server returned the HTTP status code 404 ("Not Found").

Then I tried this 

xml = Import["https://twitter.com/WolframResearch"]a

and this time I got some output. But when I continue the procedure to extract the tweets with this line

tweets = Cases[xml, XMLElement["text", _, {s_String}] :> s, ∞]

I get an empty set.

How can I import tweets into Mathematica in an effective way, given I am not a familiar with Apachi?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I think the Twitter API has changed a lot since that blog post. You'll probably need to look at the API documentation and start over. $\endgroup$
    – cormullion
    Jul 18, 2013 at 22:56
  • $\begingroup$ It is true that the Twitter API has changed and the old one is deprecated. They now require the user to be authenticated, which makes it a bit more of a job retrieving tweets. MMA has SocialMediaData which lets you authenticate yourself and then get some data, but not tweets. It would be interesting to know if there was a way of using the MMA authentication to get other things from the API. $\endgroup$
    – C. E.
    Jul 18, 2013 at 23:11
  • $\begingroup$ Using the second XML function as above, I could import some tweets but not all. And I know there is a function in R to do so. So it shoul be possible to implement it in Mathematica also. $\endgroup$
    – Morry
    Jul 19, 2013 at 8:19
  • $\begingroup$ @Morry Screen scraping can be done elegantly in a lot of languages, but it's harder in Mathematica because there aren't any really good libraries although it is absolutely possible. If you keep working with Cases, eventually you surely find a pattern that can yield all the tweets. APIs are always preferable if one can get them to work though. $\endgroup$
    – C. E.
    Jul 19, 2013 at 12:18

3 Answers 3

Reset to default
3
$\begingroup$

One first should access his developer account which is codded here as

token = HTTPClient`OAuthAuthentication[
   "ConsumerKey" -> "your_api_key", 
   "ConsumerSecret" -> "your_api_secret", 
   "RequestEndpoint" -> "https://api.twitter.com/oauth/request_token",
   "AuthorizeEndpoint" -> "https://api.twitter.com/oauth/authorize", 
   "AccessEndpoint" -> "https://api.twitter.com/oauth/access_token"];
url = "https://api.twitter.com/1.1/statuses/retweets_of_me.json";
URLFetch[url, "OAuthAuthentication" -> token]`

Then simply you should use API functions. As sample:

urlList = "https://api.twitter.com/1.1/lists/list.json?screen_name=BarackObama";

Then you should take the name of the Slug from the output and then

urlTweet = "https://api.twitter.com/1.1/lists/statuses.json?slug=ofa-legacy-conference&owner_screen_name=BarackObama&per_page=20&page=24"
xml = URLFetch[urlTweet, "OAuthAuthentication" -> token];

Then you need a simple StringCases analysis to get the tweets!

Improve this answer
$\endgroup$
1
  • $\begingroup$ You figured it out. Great answer! :) $\endgroup$
    – C. E.
    Jul 25, 2013 at 10:54
2
$\begingroup$

Perhaps it should be mentioned that since this question was active, (prior to version 10) Mathematica has introduced ServiceConnect which allows a user to access the twittersphere and handles the authentication issue. The documentation provides an example of importing tweets as the OP requested.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Could you include a valid example, for completeness? $\endgroup$
    – Kuba ♦
    Mar 8, 2019 at 12:17
0
$\begingroup$

Here's a quick hack that nearly answers the question:

xml = Import["http://twitter.com/WolframResearch", "XMLObject"];
tweet = Cases[xml, 
  XMLElement[
    "p", {"class" -> "js-tweet-text tweet-text"}, {tweet__, 
     XMLElement["a", details__]}] :> {tweet, details }, Infinity]

resulting in:

{{"Perfect your parking moves with this new Wolfram Demonstration! ", 
 {"shape" -> "rect", "class" -> "twitter-timeline-link", 
   "dir" -> "ltr", "href" -> "http://t.co/YAa2poc8qi", 
   "rel" -> "nofollow", 
   "data-expanded-url" -> "http://wolfr.am/14dZd2G", 
   "target" -> "_blank", 
   "title" -> "http://wolfr.am/14dZd2G"}, {XMLElement[
    "span", {"class" -> "invisible"}, {"http://"}], 
   XMLElement["span", {"class" -> "tco-ellipsis"}, {}], 
   XMLElement[
    "span", {"class" -> "js-display-url"}, {"wolfr.am/14dZd2G"}], 
   XMLElement["span", {"class" -> "invisible"}, {}], 
   XMLElement[
    "span", {"class" -> "tco-ellipsis"}, {XMLElement[
      "span", {"class" -> 
        "invisible"}, {" "}]}]}},
{"In case you missed it, here's our latest blog post on using 
  Mathematica to do random walks of ", 
  XMLElement[

... and so on

Looks like there's hours of 'amusement' there if you like decoding Symbolic XML!

Improve this answer
$\endgroup$
2
  • $\begingroup$ Is it possible to push the code in a way to extend the results to get the whole tweets? I mean what you proposed is just taking the first split of the tweets, not the whole history of tweets!! $\endgroup$
    – Morry
    Jul 19, 2013 at 14:27
  • $\begingroup$ I don't think they would do that. Without the Api you may be restricted. Google brings up many non-Mathematica discussions, eg this one $\endgroup$
    – cormullion
    Jul 19, 2013 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.