Extracting Certain Areas of JSON

Question

I've been working with some web scraping and have run into a wall. The website I'm working with is this: https://seekingalpha.com/symbol/TWLO/stocktalks

I am trying to scrape all of the "stock talks" that essentially look like Tweets. Using the Chrome developer tool, I've been able to trace the first "additional stock talks" as I scroll down the page. The tool gives this URL: https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=47745202&\ time=2&force_filter _symbol %5 B %5 D=twlo&force_filter _subject \ _types %5 B %5 D=StockTalks

Using the code below, I've been able to narrow down the "content" of each stock talk like so:

url = "https://seekingalpha.com/symbol/TWLO/more_stocktalks?user_id=\
47745202&time=2&force_filter_symbol%5B%5D=twlo&force_filter_subject_\
types%5B%5D=StockTalks";
ress = Association[Import[url, "JSON"]];
Keys[ress];
data = ress["cards"];
Keys[data];
content = "content" & /. data

Content is the key that all of the stock talks are located in. It seems that each scroll loads around 10 new stock talks, and they take this kind of form:

{"<p>Time To Buy Twilio (Again) $<a \
href='https://seekingalpha.com/symbol/TWLO' title='Twilio'>TWLO</a> \
<a rel='nofollow' target='_blank' \
href='https://seekingalpha.com/a/2f8xs'>https://seekingalpha.com/a/\
2f8xs</a></p>" &, 

I only want to scrape the actual text of the stock talk and not any url that is within it. For instance, for the first one I am only interested in scraping "Time to Buy Twilio (Again)", but not <a href = 'https:// ......'. .... </a> Is this at all possible? And even more so, is there a way I can automate so I can apply the script to stock talks that take the same format as above? I've been working mainly with XML so I am not sure how to work with the HTML format at all. Any help would be greatly appreciated!

Answer 1

This HTML is also valid XML, so you can continue to work with it the way that you've been working with XML before. For example

content = "content" /. data;
res = ExportString[
  ImportString[content[[1]], "XML"] /. 
   XMLElement["a", __] -> Sequence[],
  "XML"
  ]

"<p>Time To Buy Twilio (Again) $</p>"

StringReplace[res, "<p>" ~~ str__ ~~ "</p>" :> str] 
(* Or StringTrim[res, "<p>" | "</p>"] *)

"Time To Buy Twilio (Again) $"

You can wrap this up into a function if you want.

Using jsoupLink it would be something like:

Needs["jsoupLink`"]
First[ImportString[#, "HTMLDOM"]["Select", "p"]]["OwnText"] & /@ content

{"Time To Buy Twilio (Again) $", "Awful Offerings Confirmed: Acacia Communications"...