7
$\begingroup$

I want to analyze twitter data connected to a specific hashtag (#bokmässan2013). As I understand it, Twitter has recently made their complete archive of Tweets retrievable by means of API.

How can I use this API from Mathematica to get all tweets (with as much information as possible) that contains (e.g.) the tag #bokmässan2013?

$\endgroup$
  • 1
    $\begingroup$ It seems that that API requires subscription to gain authentication. If that's the case you can hardly expect anybody to test it here. An example of the use of a similar but open API (StackExchange) here $\endgroup$ – rhermans Oct 6 '15 at 16:11
6
$\begingroup$

As rhermans has noted, we can't offer support for specific APIs that have no native support in Mathematica. However one might still try to tackle the problem of retrieving the tweets that you mention. I believe there is another solution (although clearly suboptimal and not scalable), because if you search for the hashtag on Twitter then historic data will be included. Do that and then scroll down as much as you can until all tweets have been loaded into the page, then use the browser to save the HTML page to your hard drive.

To extract the content of the tweets you can use my jSoupLink` package like this:

<< jSoupLink`
jSoupLink`ParseHTML[
 "~/Desktop/bokmässan2013.html",
 ".tweet-text",
 {"text"}
 ]

This will give you a list of all tweets with that hashtag. You can change "text" to "html" to preserve the HTML code, and then use jSoupLink`ParseHTMLFragment on each tweet separately to extract more information such as links, hashtags and so on. You can also extract the author of each tweet etc. by using the appropriate CSS selectors.

If the text has non-alphabetic characters in it, that means that the text encoding is wrong. I fixed it like this:

Export[
 "~/Desktop/bokmässan2013.html",
 Quiet@Import["~/Desktop/bokmässan2013.html", "Text", CharacterEncoding -> "UTF8"],
 "Text"
 ]

The official Twitter API is supported natively in Mathematica through ServiceConnect and the related functions. Unfortunately the official Twitter API does not allow one to search for tweets that are older than a week. Also a general search is not among the built-in API requests that Mathematica offers. However, by spelunking I have figured out how to extend Mathematica's functionality to include this. I will post my code in case someone comes here looking for this type of solution, since it is very related to the question even though it can't do the requested time frame.

SearchTwitter[str_, next_: Null] := Module[{twitter, id, auth, url},
  twitter = ServiceConnect["Twitter"];
  id = OAuthClient`Private`getServiceID[twitter];
  auth = OAuthClient`Private`serviceAuthentication[id];
  If[next === Null,
   url = "https://api.twitter.com/1.1/search/tweets.json?count=100&q=" <> URLEncode[str],
   url = "https://api.twitter.com/1.1/search/tweets.json?count=100&q=" <> next
   ];
  url = First@OAuthClient`Private`getsignedurl[url, auth];

  Needs["GeneralUtilities`"];
  data = ImportString[URLFetch[url], "JSON"] // ToAssociations;

  If[KeyExistsQ[data["search_metadata"], "next_results"],
   data = <|data["statuses"], 
     SearchTwitter[str, data["search_metadata", "next_results"]]|>,
   data["statuses"]
   ]
  ]

Note that the tricky part is that you have to be authenticated in order to send this type of request to the Twitter API, and it is authentication that I'm using Mathematica for.

For example the following code will retrieve all tweets in the last seven days with the hashtag "bokmässan":

SearchTwitter["#bokmässan"]

This solution requires Mathematica 10 or later.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.