0
$\begingroup$

I'm working with some scraping and have run into a bit of a problem. The website I am scraping is this: https://seekingalpha.com/symbol/CRM/stocktalks I am essentially trying to scrape the "stock talks", but that is not the problem. Using the developer tools on my browser, I've noticed that as I scroll down the page and more Tweet like stock talks show up, the URL that initiates them look like this: https://seekingalpha.com/symbol/CRM/more_stocktalks?user_id=47745202&time=3&force_filter_symbol%5B%5D=crm&force_filter_subject_types%5B%5D=StockTalks The scroll pages are all the same except for one number that goes "&time=". In the URL above it is the third reload page, and it keeps going on and on.

Opening the first 14 of these URLs is no problem, and I simply do this with:

Import["https://seekingalpha.com/symbol/CRM/more_stocktalks?user_id=47745202&time=14&force_filter_symbol%5B%5D=crm&force_filter_subject_types%5B%5D=StockTalks","JSON"]

That opens up the JSON for the 14th website. But when I start trying to pull the 15th website and URLs beyond that, I get this error: FetchURL::httperr: The request to URL https://seekingalpha.com/symbol/CRM/more_stocktalks?user_id=47745202&time=20&force_filter_symbol%5B%5D=crm&force_filter_subject_types%5B%5D=StockTalks was not successful. The server returned the HTTP status code 403 ("Forbidden").

I've read somewhere that this might be because you can't have spaces in your urls, but I don't see the difference between the 14th URL and the 15th. Can anyone help?

$\endgroup$
  • $\begingroup$ It works for me... $\endgroup$ – C. E. Jun 30 '17 at 14:08
  • $\begingroup$ Strange, I tried it in a new notebook and it works, but doesn't when I run it on the same one as pages previous to 15 $\endgroup$ – Vic Jongmin Youn Jun 30 '17 at 16:23
  • $\begingroup$ I don't know about this particular web site, but some sites actively prevent web scraping by returning errors like 403 FORBIDDEN if too many requests are received within a short time period. $\endgroup$ – WReach Jul 1 '17 at 5:16
  • $\begingroup$ @WReach, yeah that seems to be what's happening. If I space out the scraper to different time intervals I'm able to retrieve several pages at a time $\endgroup$ – Vic Jongmin Youn Jul 3 '17 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.