1
$\begingroup$

I'm having some trouble wrapping my head around how I would go about using two arguments in one function. In the first section of the code below, I have scraped some "tweets" on the Seeking Alpha website here: https://seekingalpha.com/symbol/TWLO/stocktalks Using Google Chrome Developer Tools, I noticed that there are 30 "more stock talk" pages that are initiated as I keep scrolling down the webpage. Each page has 10 additional stock talks, which again resemble tweets on the web page. The code below scrapes all 10 stock talks that appear in the second "more stock talk" page:

getTweets[anyList_] := (
  url = "https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=2&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks";
  res1 = Association[Import[url, "JSON"]];
  Keys[res1];
  data = res1["cards"];
  content = "content" /. data;
  res = ExportString[
    ImportString[content[[anyList]], "XML"] /.
     XMLElement["a", ___] -> Sequence[],
    "XML"
    ];
  StringTrim[res, "<p>" | "</p>"]
  )
anyList = Range[10];
getTweets /@ anyList

This provides the 10 stock talks as strings in a list, which looks like this:

{"Time To Buy Twilio (Again) $", "Awful Offerings Confirmed: Acacia \
Communications and Twilio.$$", "$showing great relative strength \
today. Starting to get money from from disappointed $investors.", \
"The Loss Of Uber Is The Least Of Twilio&apos;s Worries $", "$$Uber \
Reduces Twilio Usage, Shows Why IP-Based Communications is the Modern \
Choice", "Twilio: Not All Bad $", "Lessons From Twilio&apos;s Tumble \
$", "What Next For Twilio Stock After Post Earnings Crash? Twilio Inc \
(NYSE:TWLO) $", "$falls 29% as major customer Uber pulls business. \
Our #RoboAnalyst rated TWLO Dangerous since IPO", "$Crashes On \
Earnings, Here Is The ONLY Buy Support"}

The issue I am having is applying this function to all 30 "more stock talk pages". The pattern of the URLs for these pages is rather simple, with one number changing per "more stock talk" page. The code below creates a list of all "more stock talk" pages by just changing the one variable for each page:

createList[aNumber_] := (
  firstPart = 
   StringJoin[
    "https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=", aNumber];
  secondPart = 
   StringJoin[firstPart, 
    "&force_filter_symbol%5B%5D=twlo&force_filter_subject_types%5B%5D=\
StockTalks"]
  )
rangeOfNumbers = ToString /@ Range[2, 30];
finalList = createList /@ rangeOfNumbers

This creates a list of all "more stock talk" pages' URLs, which looks like this:

{"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=2&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=3&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=4&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=5&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=6&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=7&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=8&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=9&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=10&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=11&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=12&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=13&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=14&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=15&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=16&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=17&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=18&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=19&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=20&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=21&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=22&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=23&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=24&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=25&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=26&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=27&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=28&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=29&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=30&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks"}

I can run each of the 30 or so URLs in the first function separately, but also feel there must be way that I can map the function into the list of all URLs. I've researched how to use functions with two arguments, but am still having some trouble. Any help would be greatly appreciated!

$\endgroup$
  • $\begingroup$ Could you provide a minimal example of function and expected output? Feels like there is too much noise which makes the core of the problem obscure. $\endgroup$ – Kuba Jun 27 '17 at 13:26
  • $\begingroup$ I meant, do we need all those long urls etc? Can't you mimic the idea with an artificial input and function? $\endgroup$ – Kuba Jun 27 '17 at 13:38
  • $\begingroup$ @Kuba I added the outputs of the two functions in the question. My desired output would be to have a function that outputs all the stock talks in one list --- so for example instead of having just 10 as in the output from the first function above, it would ideally have around 300. $\endgroup$ – Vic Jongmin Youn Jun 27 '17 at 13:39
  • $\begingroup$ Yes that's kind of what I was asking about, which would require two arguments right? One to crawl through the 30 or so URLs and another to get all 10 stock talks per page? I've tried ImportString[content[[Range[10]]], "XML"] /. but that doesn't seem to work $\endgroup$ – Vic Jongmin Youn Jun 27 '17 at 13:40
  • $\begingroup$ It is likely I am misunderstanding the issue at large :/ $\endgroup$ – Vic Jongmin Youn Jun 27 '17 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.