1
$\begingroup$

I'm having some trouble wrapping my head around how I would go about using two arguments in one function. In the first section of the code below, I have scraped some "tweets" on the Seeking Alpha website here: https://seekingalpha.com/symbol/TWLO/stocktalks Using Google Chrome Developer Tools, I noticed that there are 30 "more stock talk" pages that are initiated as I keep scrolling down the webpage. Each page has 10 additional stock talks, which again resemble tweets on the web page. The code below scrapes all 10 stock talks that appear in the second "more stock talk" page: 

getTweets[anyList_] := (
  url = "https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=2&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks";
  res1 = Association[Import[url, "JSON"]];
  Keys[res1];
  data = res1["cards"];
  content = "content" /. data;
  res = ExportString[
    ImportString[content[[anyList]], "XML"] /.
     XMLElement["a", ___] -> Sequence[],
    "XML"
    ];
  StringTrim[res, "<p>" | "</p>"]
  )
anyList = Range[10];
getTweets /@ anyList

This provides the 10 stock talks as strings in a list, which looks like this:

{"Time To Buy Twilio (Again) $", "Awful Offerings Confirmed: Acacia \
Communications and Twilio.$$", "$showing great relative strength \
today. Starting to get money from from disappointed $investors.", \
"The Loss Of Uber Is The Least Of Twilio&apos;s Worries $", "$$Uber \
Reduces Twilio Usage, Shows Why IP-Based Communications is the Modern \
Choice", "Twilio: Not All Bad $", "Lessons From Twilio&apos;s Tumble \
$", "What Next For Twilio Stock After Post Earnings Crash? Twilio Inc \
(NYSE:TWLO) $", "$falls 29% as major customer Uber pulls business. \
Our #RoboAnalyst rated TWLO Dangerous since IPO", "$Crashes On \
Earnings, Here Is The ONLY Buy Support"}

The issue I am having is applying this function to all 30 "more stock talk pages". The pattern of the URLs for these pages is rather simple, with one number changing per "more stock talk" page. The code below creates a list of all "more stock talk" pages by just changing the one variable for each page:

createList[aNumber_] := (
  firstPart = 
   StringJoin[
    "https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=", aNumber];
  secondPart = 
   StringJoin[firstPart, 
    "&force_filter_symbol%5B%5D=twlo&force_filter_subject_types%5B%5D=\
StockTalks"]
  )
rangeOfNumbers = ToString /@ Range[2, 30];
finalList = createList /@ rangeOfNumbers

This creates a list of all "more stock talk" pages' URLs, which looks like this:

{"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=2&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=3&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=4&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=5&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=6&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=7&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=8&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=9&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=10&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=11&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=12&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=13&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=14&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=15&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=16&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=17&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=18&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=19&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=20&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=21&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=22&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=23&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=24&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=25&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=26&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=27&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=28&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=29&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks", \
"https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=30&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks"}

I can run each of the 30 or so URLs in the first function separately, but also feel there must be way that I can map the function into the list of all URLs. I've researched how to use functions with two arguments, but am still having some trouble. Any help would be greatly appreciated!

Improve this question
$\endgroup$
6
  • $\begingroup$ Could you provide a minimal example of function and expected output? Feels like there is too much noise which makes the core of the problem obscure. $\endgroup$
    – Kuba ♦
    Jun 27, 2017 at 13:26
  • $\begingroup$ I meant, do we need all those long urls etc? Can't you mimic the idea with an artificial input and function? $\endgroup$
    – Kuba ♦
    Jun 27, 2017 at 13:38
  • $\begingroup$ @Kuba I added the outputs of the two functions in the question. My desired output would be to have a function that outputs all the stock talks in one list --- so for example instead of having just 10 as in the output from the first function above, it would ideally have around 300. $\endgroup$
    – Vic Jongmin Youn
    Jun 27, 2017 at 13:39
  • $\begingroup$ Yes that's kind of what I was asking about, which would require two arguments right? One to crawl through the 30 or so URLs and another to get all 10 stock talks per page? I've tried ImportString[content[[Range[10]]], "XML"] /. but that doesn't seem to work $\endgroup$
    – Vic Jongmin Youn
    Jun 27, 2017 at 13:40
  • $\begingroup$ It is likely I am misunderstanding the issue at large :/ $\endgroup$
    – Vic Jongmin Youn
    Jun 27, 2017 at 13:42

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.